





Military Operations Research Society 


Di E 


MILITARY OPERATIONS RESEARCH SOCIETY 





Command, Control, Communications, 
Intelligence, Electronic Warfare 
Measures of Effectiveness 













| 3 
euren)  (CIEW MOE) 
| ter pube aad Wor kshop 


Bes) 


Vernon M. Bettencourt, Jr—General Chair 


Final Report 
Edited by LTC Thomas J. Pawlowski III 


Contributors: 
Mr. James B. Duff 


| 00 Lt Col Kenneth C. Konwin 
1021 Q ( 01 Mr. Donald W. Kroening 
Dr. Michael G. Sovereign Lie o; ips "— 
| Dr. Stuart H. Starr ALCORTA g 


Conducted at Fort Leavenworth, Kansas 


20-23 October 1992 


101 South Whiting Street € Suite 202 € Alexandria, VA 22304-3483 e (703)-751-7290 € FAX: (703)-751-8171 
e-mail: rwiles@dgis.dtic.dla. mil 








DISCLAIMER 


This Military Operations Research Society workshop report faithfully summarizes the findings 
of a three-day meeting of experts, users, and parties interested in the subject area. While it is 
not generally intended to be a comprehensive treatise on the subject, it does reflect the major 
concerns, insights, thoughts, and directions of the authors and discussants at the time of the 
workshop. 











REPORT DOCUMENTATION PAGE | 


Public reporong burden for ths collection of information is esumated to average 1 hour per responsa, including the ume for renewing instructions, seerching exstng dete sources, 
gathenne and mamtaming the dete needed, and completing and reviewing the collecnon of informanon. Send comments regarding this burden esumate or any other sapect of ths 
collecnon of information, inctuding suggestions for reducing ts burden, to Washington Headquarters Services, Directorate for information Operanone and Reporta, 1215 Jefferson 
Cavs Highway, Suite 1204. Arlington, VA 22202-4302. and to the Office of Management and Budget. Paperwork Reducuon Proyect (0704-0188), Washington, DC. 20603. 


1. AGENCY USE ONLY (Leave blank) | 2. REPORT DATE 3. REPORT TYPE AND DATES COVERED 
ne A oy Imm 
4. TITLE AND SUBTITLE 5. FUNDING NUMBERS 
.Command, Control, Communications, intelligence, 
Electronic Warfare Measures of Effectiveness O&MN 
(C3IEW MOE) Workshop 


6. AUTHOR(S) 























LTC Thomas J. Pawlowski III, Editor 










7. PERFORMING ORGANIZATION NAME(S) AND ADORESS(ES) 


Military Operations Research Society 
101 S. Whiting Street, Suite 202 
Alexandria, VA 22304 


8. PERFORMING QRGANIZATION 
REPORT NUMBER 


C3IEW MOE Workshop 
Report 



















3. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 


Office of Naval Research 
Arlington, VA 22217 and 

OCNO, N811, The Pentagon 
Washington, DC 20350-2000 


11. SUPPLEMENTARY NOTES 


10. SPONSORING/MONITORING 
AGENCY REPORT NUMBER 




















12a. DISTRIBUTION/AVAILABILITY STATEMENT 


Distribution Statement A: Unlimited. 
Approved for Public Release. 


12b. DISTRIBUTION CODE 








13. ABSTRACT (Maximum 200 words) 







The purpose of this report is to document the results of 
a MORS sponsored workshop held at Fort Leavenworth, Kansas, 
in October 1992 to address the development and application 
of CJIEW MOE throughout DoD. 












14. SUBJECT TERMS 





15. NUMBER OF PAGES 
60 
16. PRICE CODE 


19. SECURITY CLASSIFICATION | 20. LIMITATION OF 
OF ABSTRACT ABSTRACT 
Unclassified Unclassified Unclassified Unlimited 


NSN 7540-01-280-5500 Standard Form 298 (Rav. 2-89) 
Prescribed by ANSI Sta. 239-18 







17. SECURITY CLASSIFICATION 
OF REPORT 


18. SECURITY CLASSIFICATION 
OF THIS PAGE 










Mo perm APTE aem mein EE Cue a o 4, aS. aA s. Ax AN » . NIU AT AE A S ADLER nd M AMA. u _ 











Military Operations Research Society 


Dni» E 


MILITARY OPERATIONS RESEARCH SOCIETY 





Command, Control, Communications, 
Intelligence, Electronic Warfare 
Measures of Effectiveness 
(CIEW MOE) 

Workshop 


Mr. Vernon M. Bettencourt, Jr—General Chair 


Final Report 
Edited by LTC Thomas J. Pawlowski III 


Contributors: 
Mr. James B. Duff 
Lt Col Kenneth C. Konwin 
Mr. Donald W. Kroening 
Dr. Michael G. Sovereign 
Dr. Stuart H. Starr 


Conducted at Fort Leavenworth, Kansas 


20-23 October 1992 


101 South Whiting Street € Suite 202 € Alexandria, VA 22304-3483 € (703)-751-7290 € FAX: (703)-751-8171 
e-mail: rwiles@dgis.dtic.dla.muil 











The Military Operations Research Society 


The purpose of the Military Operations Research Society is to enhance the quality and 
effectiveness of classified and unclassified military operations research. To accomplish this 
purpose, the Society provides media for professional exchange and peer criticism among 
students, theoreticians, practitioners, and users of military operations research. These media 
consist primarily of the traditional annual MORS symposia (classified), their published 
proceedings, special mini-symposia, workshops, colloquia and special purpose monographs. The 
forum provided by these media is directed to display the state of the art, to encourage consistent 
professional quality, to stimulate communication and interaction between practitioners and users, 
and to foster the interest and development of students of operations research. In performing its 
function, the Military Operations Research Society does not make or advocate official policy nor 
does it attempt to influence the formulation of policy. Matters discussed or statements made 
during the course of its symposia or printed in its publications represent the positions of the 
individual participants and authors and not of the Society. | 


The Military Operations Research Society is operated by a Board of Directors consisting of 
30 members, 28 of whom are elected by vote of the Board to serve a term of four years. The 
persons nominated for this election are normally individuals who have attained recognition and 
prominence in the field of military operations research and who have demonstrated an active 
interest in its programs and activities. The remaining two members of the Board of Directors 
are the Past President who serves by right and the Executive Director who serves as a 
consequence of his position. A limited number of Advisory Directors are appointed from time 
to time, usually a 1-year term, to perform some particular function. Since a major portion of 
the Society's affairs is connected with classified services to military sponsors, the Society does 
not have a general membership in the sense that other professional societies have them. The 
members of MORS are the Directors, persons who have attended a MORS meeting within the 
past three years and Fellows of the Society (FS) who, in recognition of their unique contributions 
to the Society, are elected by the Board of Directors for life. 


MORS is sponsored by: 

e The Deputy Under Secretary of the Army (Operations Research) 

e The Director Assessments Division, Office of the Chief of Naval Operations 

e The Director of Modeling, Simulation and Analysis, Deputy Chief of Staff, Plans and 
Operations, Headquarters, US Air Force 

e The Commanding General, Marine Corps Combat Developments Command 

The Director of Force Structure, Resource and Assessment, The Joint Staff 

e The Director Program Analysis and Evaluation, Office Secretary of Defense 


vi 











TABLE OF CONTENTS 


Chapter I - Introduction ....... eel mI t tcr l 
10: ¿Purpose iria ad Nos dde A ee A xc iie pede deg l 
1.1 Background ................... et aaa 1 
1.2 Workshop Organization ........ +... e... ilc 2 
1.3 Workshop Objectives ........o.ooooo. RÀ À e t 6 tr 3 
Chapter II - Definition of Force-level CIEW ... 1... 2... eee ee n 5 
2-0- Introduction: + 9 RC a Edo cR nr PUE ee wa 3 
2.1 The Role of CIEW ina Campaign ........... o... . e... e... 5 
222 Differences in Terminology ............... +... I IR Be 6 
2.3 Essential Elements of Analysis ......... +... «o... n ne 6 
Chapter III - Description of Force-level CIEW Measures .......ooooooooo mo... 9 
3.0: Introduction: «¿arre A A AS ah 9 
31 SDelintions ea RICE ii E O AA a AAA 9 
3.2 Measure Relationships ............. +... e... e. ra 10 
3.3 Differences in Perspectives of Measures ....... ooo... . eee ee +... 11 
3.4 Methodologies for Deriving Measures ...... ooo... +... eee 12 
Chapter IV - Quantifying CIEW Measures... 2... 2-2. eee nn 15 
4.0 Current Capabilities ............. «+... RR I I +. 15 
4.1 Problems with Current Capabilities .......... +... eee 17 
42: Solution. to Problems << =... 0 ob e A SES an a A 18 
Chapter V - Tools and Methodologies for CIEW Evaluation ...........-+.-.-- 19 
5.0 Deficiencies in Tools and Methodologies ................. eese 19 
5.1 Overcoming Current Deficiencies .......... ooo... nn nn 19 
5.2 Improving Our Tools and Methodologies ........... eene 20 
Chapter VI - Findings and Recommendations ...... oo... om. n n nn 21 
ROMS RUOREESRITSIIOIOOILLITILI LLLILIDR LEO LE 21 
6.1 Recommendations ...... eee ehh hn n nnn 23 
62 Summary ice o x acd nee dob HC an SE doo e de dormi t 24 
APPENDICES 
A Excerpts from Workshop Dinner Speech ........ ooo. eee eee eee 25 
B Excerpts from Keynote Address ......... oo... errar. 31 
C Excerpts from Objective 2 Plenary Address ...... +... o... o... eee: 35 
D Excerpts from Objective 3 Plenary Address ........ lene 39 
E Excerpts from Objective 4 Plenary Address ......... ooo. oo... .> 43 
F Planning and Organizing Committee ........... e n n nn 51 
G Working Group Chairs, Co-Chairs and Speakers ........... . +... > 53 
H Workshop Agenda .......... e rer 55 
D. Partieipants «2 ase da ARS E Ree dodo AA 57 
MN nO u: se TOIT ILL E AA AA A 59 


vii 





FIGURES 
II-1 ‘Onion’ Diagram .............. o a al ee ren 10 
II-2 Hierarchal Relationship and Measures ........ e... o... +... +... 11 
IV-1 C’FAM Methodology ........... eR IRA IRA c4 15 
TABLES 
I1 | Workshop Objectives ......... eee I rn ia O 
II1 Essential Elements of Analysis for CIEW ........... llle 7 
HI-1 Levels of a System Property ......... le ne 11 
H-2 Areas of Contribution for a CIEW System ......... e... o... +... 13 
IV-1 OPVIEW Methodology ........ «o... . «o... IR t nn 16 


viil 














Chapter I 


Introduction 


1.0 Purpose 

The requirement to measure the effects 
of C’IEW systems on battle outcome contin- 
ues to challenge the operations research 
community. This community has long recog- 
nized the difficulty in measuring C’IEW 
system effectiveness in terms of battlefield 
attrition and end strengths. However, with 
the recent political changes in the world, the 
changing role of US military to a power 
projection force, and the lessons of Desert 
Storm, the need to analyze and understand 
the effects of CÍTEW systems has never been 
greater. The doctrinal evolution of military 
operations has heightened the need for cam- 
paign level C’IEW measures of effectiveness 


(MOE). 


The purpose of this report is to docu- 
ment the results of a MORS sponsored 
workshop held at Fort Leavenworth, Kan- 
sas, in October 1992 to address the develop- 
ment and application of C’IEW MOE 
throughout DoD. 


1.1 Background 

Research efforts directed at the develop- 
ment and application of C’IEW measures of 
effectiveness have experienced periodic 
surges in interest and activity followed by 
relatively dormant periods. For example, 
within the U.S. Army, the period from the 
mid- to late-1970s saw development of new 
automated command and control systems, 
such as the Tactical Fire Direction System 
(TACFIRE) and the Tactical Operations 
System (TOS). These systems were being 
developed with all haste and they held great 


promise for increasing force effectiveness. 
Along with these developments, of course, 
came the requirements for operational test 
and evaluation and for the conduct of Cost 
and Operational Effectiveness Analysis 
(COEA). This led to a flurry of activity to 
develop C’IEW measures of effectiveness 
Unfortunately, the tests and the COEAs 
failed to show an increase in effectiveness 
which could be attributed to these systems. 
This was due, in part, to the inability to 
measure the effectiveness of these systems 
either in the tests or in the combat models 
used in the COEAs. 


This inability to measure effectiveness 
contributed significantly not only to the 
cancellation of the TOS program by Con- 
gress, but also to a marked decline in the 
Army's efforts to develop C’IEW measures 
of effectiveness. 


By 1985 the rapid growth of computer 
technology had greatly expanded the capabil- 
ities of automated C’IEW systems. Com- 
mand and control systems became larger, 
more automated, and more integrated with 
communications, intelligence, and electronic 
warfare systems. In light of these develop- 
ments, there was a renewed interest in the 
development of a practical set of measures 
of effectiveness and evaluation methodolo- 
gies to support military decision makers 
throughout the DoD in their assessments of 
command and control and C’IEW systems. 
Recognizing this need, MORS sponsored a 
C? Evaluation Workshop in January 1985 to 
develop a conceptual model for C? and 








provide an analysis framework for measur- 
ing the performance and effectiveness of C* 
systems within that model. This framework 
was called the Modular Command and Con- 
trol Evaluation Structure (MCES). The 
product of the workshop was a draft report 
which was subsequently finalized and pub- 
lished by MORS in June 1986. Also in 
1986, the application of MCES to a series of 
military C’IEW systems was demonstrated 
and results published by the Naval Postgrad- 
uate School in September 1986. From this 
point through 1991 few new developments 
of C’IEW measure of effectiveness are 
documented. 


In November of 1911, the US Army 
TRADOC Analysis Command (TRAC), at 
the direction and under the sponsorship of 
Mr. Walt Hollis, Deputy Under Secretary of 
the Army for Operations Research 
[DUSA(OR)], conducted a C’I Modeling 
and Simulation Conference. The purpose of 
this conference was to share what C'I activi- 
ty was in progress in the Army training, 
testing, analysis, and research communities 
and to determine what needed to be done to 
collectively improve these activities. One 
major recommendation from this conference 
was that more work was needed in C'I mea- 
sures of effectiveness development and that 
this work needed to result in more consistent 
measures across these communities. As a 
result of this conference recommendation, 
Mr Hollis, as a MORS sponsor, asked 
MORS officials to consider sponsoring a 
workshop to address C’IEW MOE develop- 
ment. TRAC volunteered to host this work- 
shop for MORS at Fort Leavenworth. The 
workshop was conducted from 20 through 
23 October 1992 and was attended by some 
one hundred trainers, testers, analysts, and 
researchers from throughout DoD and the 
civilian community. 


IND 
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Workshop Organization 

The attendees were divided into four 
working groups which reflected the function- 
al areas of the military OR community: 
Operations, Plans, Training; Analysis; Test 
and Evaluation; and Research. Additionally, 
a synthesis group participated in the discus- 
sions with all working groups. The synthe- 
sıs group had a member in each working 
group to identify common threads among the 
groups and to summarize the main points of 
the discussions. 


The workshop began the evening of 20 
Oct 92 with a dinner and an address by 
COL Jim McDonough, Director of the 
School of Advanced Military Studies, 
USACGSC. COL McDonoush discussed the 
concept of a campaign and provided some 
insights on the changes being made to the 
Army’s doctrine. The intention of his dis- 
cussion was to focus the group on relating 
MOP, MOE and MOFE to the theater cam- 
paign level. Appendix A provides excerpts 
from this presentation. 


The workshop ran as a series of four 
working sessions, with each session discuss- 
ing one of the four objectives. Each session 
began with an address by an invited speaker. 
After each address, the participants divided 
into their working groups to address the 
appropriate objective. At the end of the 
working session, the participants assembled 
to hear each Working Group Chair report on 
the findings of their working group. 


The keynote address, which also acted as 
the address for the first session, was pre- 
sented by LTG Wilson A. Shoffner, Com- 
mander, USA Combined Arms Command. 
General Shoffner shared his views on com- 
mand and control, and the effect of recent 
world changes on the conduct of campaigns. 


General Shoffner provided a superb discus- 
sion of the dynamics of the future battle- 
field, the role of battle command in a cam- 
paign, and the relationship between com- 
mand and control. He emphasized that 
command and control are not one word. 
Command is an art that is the commander's 
business, while control is a science that is 
the staff's business. A key issue that he 
raised was to give up on modeling the intu- 
itive aspect of the decision making process 
and use a man-in-the-loop to represent the 
decision maker. Appendix B contains ex- 
cerpts from LTG Shoffner's presentation. 


The other plenary presentations were 
made by Dr. Mike Sovereign, Dr. Hank 
Dubin, and Mr. Ed Brady. Dr. Sovereign 
reviewed MCES, which was developed 
during the 1985 MORS Workshop. He used 
the VTOL UAV asn example to show 





how to implement MCES on a real sys- 
tem.Excerpts from his presentation are 
located in appendix C. Dr. Dubin addressed 
the definitional context of C^I measures, 
typical measures, teaching command and 
control, and the challenges associated with 
this field. Highlights of his presentation are 
in appendix D. Mr. Brady's address provid- 
ed an historical perspective with the key 
trends and key issues, and presented some 
thoughts on emerging tools and technolo- 
gies. Excerpts are in appendix E. 


1.3 Workshop Objectives 

There were four objectives for the work- 
shop. Each working group was asked to 
address the questions associated with these 
objectives. The objectives and questions are 
listed in the Table I-1 below: 


Table 1-1. Workshop Objectives 





Objective 1: Definition of Force-level C’IEW 





Objective 2: 


ers and developers? 


e What is the nature of force-level C'IEW within a Campaign? 

e What are terminology differences among services? 

e What are key essential elements of analysis (EEA) associated with force-level C'IEW? 
Description of Force-level C°IEW Measures 

e What are C'IEW MOP, MOE and MOFE and the relationship among them? 

e What are the differences in perspective between testers, trainers, analysts, research- 


e What approaches/methodologies can be used to develop these measures? 


e How can testers, trainers, analysts, researchers and developers best work together? 


Objective 3: Quantifying C'IEW Measures and Answering C'IEW EEA 
e What Capability currently exists to collect and quantify C'IEW measures? 
€ What are the problems associated with this capability and how can they be over- 
come? 
Objective 4: Tools and Methodologies to support C'IEW Evaluation 


e What deficiencies exist in the tools and methodologies used to measure C^IEW? 
e How can emerging technology overcome these deficiencies? 
eo How can we work together to improve our tools and methodologies? 































Chapter II | 


Definition of Force-level C’IEW 


2.0 Introduction 

To proceed through the process of de- 
scribing force-level C’IEW measures of 
effectiveness, it is important to first define 
what is meant by a campaign and force-level 
C’IEW. The working groups identified nu- 
merous descriptions from a multitude of 
perspectives. One description for a campaign 
from the draft Joint Pub 5-00.1 says that 
"campaigns represent the art of linking bat- 
tles and engagements in an operational 
design to accomplish strategic objectives..." 
In this context a campaign can then be 
defined as "a series of related military oper- 
ations aimed to strategic objectives within a 
given time and space." 


With the role of the military focusing 
more on operations other than war, i.e., 
peacekeeping activities, relief activities such 
as Operation Provide Comfort, and the relief 
effort in Somalia, the characteristics of a 
campaign may change and likewise change 
the nature of C’IEW within a campaign. As 
noted in COL McDonough's dinner speech 
(see appendix A), the Army is rewriting its 
doctrine to incorporate the entire spectrum 
of military operations, from peacekeeping 
activities to total war. Any effort in describ- 
ing C’IEW measures of effectiveness must 
ultimately account for the variety of mis- 
sions that can be undertaken by the military 
in a post-Cold War environment. However, 
for purposes of narrowing the scope, this 
workshop focused on combat operations 
with forces already in-theater. 


2.1 The Role of C’IEW in a Campaign 
A campaign provides a framework for 
discussing force-level C’IEW. Today, any 
campaign conducted by U.S. military forces 
must be considered a joint campaign run by 
a CINC or Joint Task Force Commander. 
Single service campaigns are not a reality 
any more. In terms of unit size, force-level 
implies corps or echelons above corps 
(EAC) for Army, numbered Air Force, and 
Battle Group or Battle Force for the Navy. 


Joint Pub 1-02 defines command and 
control as ". . . the exercise of authority and 
direction by a properly designated com- 
mander over assigned forces in the accom- 
plshment of a mission. Command and 
control functions are performed through an 
arrangement of personnel, equipment, com- 
munications, facilities, and procedures 
employed by a commander in planning, 
directing, coordinating, and controlling 
forces and operations in the accomplishment 
of the mission." This definition is invariably 
included at the beginning of any document 
discussing command and control. This 
report will be no different in that sense. 
What is important from this definition is the 
understanding of the role command and 
control systems in supporting the command- 
er to accomplish these functions. 


From Control of the Joint Force: A New 
Perspective, "General Depuy reminds us that 
the purpose of command and control is to 
produce unity of effort from a diversity of 
means and that the key ingredient is not the 
automated systems, but the mind of the 











commander.” The focus of command and 
control, and all of C’IEW, must be to better 
support the friendly commander's decision 
making process and to degrade that of the 
opposing commander. To make the best 
decision, the commander must have the best 
information available. It 1s the role of the 
C’IEW to provide that relevant information 
in a timely, accurate manner. 


In his keynote address (Appendix B), 
LTG Shoffner described the nature of the 
future battlefield as having the potential for 
an accelerated tempo. Lethality is increas- 
ing; mobility is increasing; weapons preci- 
sion 1s increasing; and, range of fires 1s 
increasing. The acceleration in these battle- 
field dynamics brings an increase in the 
amount of information and intelligence that 
the commander must comprehend in a short 
period of time. The employment of multiple 
lethal weapons in the attack will require 
intricate coordination and timing that only a 
sophisticated command and control system 
can provide. Likewise, disruption of the 
enemy's capability could allow the com- 
mander to control or slow the op tempo 
when beneficial to friendly forces. 


While command is primarily an art, con- 
trol is primarily a science. Intelligence pro- 
vides the perception of the enemy, while 
electronic warfare provides the means to 
deny the enemy an accurate perception of 
friendly forces. Communications provides 
the glue to make it all tie together. C’IEW 
as a whole provides the grease to reduce the 
friction of war, across concept formulation, 
planning, and execution. 


2.2 . Differences in Terminology 

One of the hurdles that must be over- 
come in the effort of describing C’IEW and 
C’IEW measures is the difference in termi- 





nology among the services and the tremen- 
dous number of acronyms. There are real . 
differences in C? systems among the services 
as well as different Intelligence and EW sys- 
tems, so terminology can not be expected to 
be the same. Message text formats and data 
dictionaries are becoming standardized but 
procedures are still unique, often to specific 
commands and commanders. This affects the 
problem of interoperability in both the real 
and the modeling worlds. 


No adequate joint theater-level campaign 
C’IEW model exists although PACOM may 
be beginning to build one. Joint exercise 
support systems are being cobbled together: 
Aggregate Level Simulation Protocol 
(ALSP) and Data Bus, for example, are 
linkages of major war games that could be 
used to examine C’IEW. 


2.3 Essential Elements of Analysis 

Not only are there differences among the 
services in terminology of C’IEW systems, 
there are also differences among the services 
in analysis terminology. The term, Essential 
Elements of Analysis (EEA), is not often 
used outside of the Army. One suggested 
term is “critical analysis questions” as an 
equivalent, but there is difficulty even in 
defining those. One question which recurs is 
whether EEA are exclusively developed to 
support analysis for acquisition or for opera- 
tions, training, etc. Army EEA seem to be 
designed for use across acquisition of all 
types of systems. For C’IEW analysis, the 
proposed essential elements of analysis are 
summarized in Table II-1. These emphasize 
the key decisions that are made in the cam- 
paign and the information required to sup- 
port the decisions. Unfortunately, there is no 
reference document for the campaign deci- 
sions! 





Table Il-1. Essential Elements of Analysis for C’IEW 












When is information needed? 






The view of C’IEW EEA shown in 
Table 2-1 suggests that the CÍTEW process 
Is a hierarchy with command at the top, 
control in the middle, and communications 
tying sensors to information, intelligence, 
and EW systems at the bottom. With this 
structure, some EW functions, such as 
jammers and radar warning receivers on 
aircraft, reside outside of a force-level 
discussion. 


What is the applicability of information at each echelon? How does a system get the right da- 
talinformation to the right person at the right time? 


Does missing, late or inaccurate information change key decisions? 





e What are system classification issues that need to be overcome to complete analysis? 
e What does a C'IEW system need to do and how well does it need to do it? 

e What are the key decisions that affect the campaign outcome? 

e Does the proposed change/system/process support the commander's needs? 


e What are the key information elements needed by the joint component commanders and their statis? 


An important question that remains 
unanswered is: at what level of resolution 
should the decisions be represented in an 
analysis of force-level C’IEW? No concur- 
rence was reached with some believing that 
an aggregate level was adequate, while 
others believed that much more detailed 
modeling would be needed, for example that 
logistics decisions would be necessary. 











Chapter HI 


Description of Force-level CTEW Measures 


3.0 Introduction 

The MORS workshop on C* MOE in 
1985 laid the foundation for much of the 
content of this chapter. One of the results 
of that workshop was the Modular C? Evalu- 
ation Structure (MCES), a methodology to 
identify measures that are appropriate for C* 
analysis. The MCES approach was de- 
scribed in a plenary presentation by Dr. 
Mike Sovereign. His presentation 1s summa- 
rized in Appendix C. Key elements of the 
work from the 1985 workshop include a set 
of definitions for measures and the establish- 
ment of a relationship among those mea- 
sures. This relationship provides the frame- 
work in which an analysis of a C^ system 
can occur. 


3.1 Definitions 

The 1985 workshop provided the initial 
set of definitions for C^ measures. Ingram 
and Short modified these definitions slightly 
in their 1991 paper, Command and Control 
Measures: A Proposed Approach. These 
definitions are modified and extended to 
apply to C’IEW systems: 


CIEW system. A C’IEW system has 
three components: physical entities, struc- 
tures, and processes. 


(1) Physical entities refer to equip- 
ment, software, facilities, and people. 


(2) Structures identify the arrange- 
ment and interrelationships of physical 
entities, procedures, protocols, concepts of 
operation, and information patterns. Such 


arrangements are often spatial and temporal. 


(3) Processes are what the system is 
doing and reflect the functions carried out 
by the system. 


Dimensional parameter. A property or 
characteristic inherent in the physical entities 
whose value determines system behavior and 
the structure under question even when at 
rest. Examples are size, weight, capacity, 
and number of pixels. These measures are 
obviously scenario independent. 


C'IEW measure of performance (C’IEW 
MOP). A measure of an attribute of system 
behavior. Examples are throughput and 
error rate. CIEW MOP are related to the 
inherent parameters (physical and structur- 
al), are internal to the system being ana- 
lyzed, and are scenario-independent. 


C’IEW measure of effectiveness (CIEW 
MOB). A measure of how a C’IEW system 
affects the other entities within an operation- 
al environment. Examples are reaction time, 
number of targets detected, and susceptibili- 
ty to deception). CIEW MOE are measured 
relative to some standard or baseline, which 
is often implicit (e.g., how a perfect C’ 
system would perform). C’IEW MOE are 
scenario-dependent. 


Measure of force effectiveness (MOFE). 
A measure of how the force performs its 
mission. These are typically measures that 
are used for weapon system analysis, for 
example, loss exchange ratios (LERs), or 








number of weapons destroyed. As with 
C’IEW MOE, MOFE are 
scenario-dependent. 


Boundary of a CIEW system. The delin- 
eation between the C’IEW system being 
studied and the environment. Within this 
definition, measures can shift from one type 
to another depending on the context of the 
C’IEW system boundary. Figure III-1 graph- 
ically shows the relationship of the measures 
defined above. This diagram is typically 
referred to as the ‘onion’ diagram, because 
each layer can be pealed back to reveal the 
next lower level. 


3.2 Measure Relationships 

NOTE: For purposes of generalization, the 
term, Measures of Merit (MoMs), will be 
used to refer to all measures. 


There are several ways to view C’IEW 
Measures of Merit and their relationship. 
Each viewpoint provides a different perspec- 
tive toward the measures. Three viewpoints 
are presented here: the 'onion' diagram, the 
hierarchical relationship, and the properties 
measures. 


The 'onion' diagram of the MCES can 
be used as a tool for bounding the problem. 
But, the boundaries of the system must be 
drawn carefully for the definitions to work. 
That must be accomplished within the con- 
text of a particular analysis to answer the 
EEA. The perspective from which the analy- 
sis occurs may change a measure of effec- 
tiveness to a measure of performance. For 
example, depending on the level of the 
analysis to be examined, measures which are 
normally MOEs for lower level examina- 


Environment 


Force Bound 





Dimension 
Performance 
Effectiveness 

Force Effectiveness 


— 
= 
= 
— 
— 
— 
— 


Figure Ill-1. ‘Onion Diagram’ 
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tions may become assumptions at higher 
levels. 

The hierarchical relationship of  mea- 
sures of merit suggests an ordered structure 
that allows the linking of lower level mea- 
sures to higher level measures. In this rela- 
tionship, measures of effectiveness (MOE) 
can be linked to certain measures of perfor- 
mance (MOP) which, in turn, can be linked 
to specific dimensional parameters (DP). To 
use this relationship effectively, the lower 
measures can be analyzed to provide some 
indication of the higher level measures. If a 
direct relationship can be established be- 
tween lower and higher measures, conclu- 
sions can be reached about system effective- 
ness through the analysis of system perfor- 
mance. Care must be taken to appropriately 
extend a cause and effect relationship to this 
hierarchical structure. Figure III-2 depicts 
the hierarchical relationship for both weapon 
systems and C’IEW systems. 





Figure Ili-2. Hierarchical Relationship 
of Measures 


Measures can be described in terms of 
the level of resolution of a particular proper- 
ty of the CÍTEW system. This approach dis- 
sects a system in terms of its functionality or 
purpose and the properties associated with 
that functionality. Through this description, 
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measures of merit can be developed and 
associated with the key properties of the 
system. Table III-1 provides an example of 
the levels of a system property. Although 
the term 'MOE' is used in an analysis, it is 
often unclear which property level is being 
considered. 


Table lll-1. Levels of a System Property 





PROPERTY EXAMPLE 

1. Family Name Vulnerability 

2. Surname Susceptibility to 
Jamming 

3. System Reference VAV Data Link 

4. Functional Ref. Sense 

5. Units BITS/Sec 

6. Value 1000 

7. 'Threshold I 


3.3Differences in Perspectives of Measures 
There are clearly differences in the perspec- 
tive of the researchers, analysts, testers and 
trainers. This is due in part to the differenc- 
es in disciplines, but it can also be attributed 
to the different stages of acquisition in 
which communities interact with the CTEW 
system. Researchers typically function in the 
long term, developing several new technolo- 
gies for later application in a system. Devel- 
opers are in the mid to long term in devel- 
oping requirements of a few potential sys- 
tems. Testers perform their evaluations of 
one or two systems in the near to mid term 
as a system reaches its later milestones prior 
to fielding. Trainers operate with one system 
in the near term when the fielding com- 
mences. Analysts operate across the entire 
range working with researchers to imple- 
ment new technologies in analytic models, 
supporting the developer by analyzing sever- 





al concepts, supporting the acquisition pro- 
cess in general with COEAs and weapon 
system versus C'I system tradeoff analyses. 


The difference in time horizon and the 
separation of the communities make it diffi- 
cult to coordinate the development of mea- 
sures. However, it is not clear that this must 
be done except for neatness. The EEA are 
not necessarily the same for each communi- 
ty. That is not to say that it would not be 
valuable for each community to pay atten- 
tion to what measures are used by the oth- 
ers. With the trend toward compressing 
these timelines, the need for cooperation 
among the communities appears to be grow- 


ing. 


There are three points that summarize 
the concerns of the workshop: 


e The COEA process is putting pressure 
on early identification of measures and 
linkage of analysis measures to testing 
measures. 


e Measures for COEAs must enable trade- 
offs to be made between C’IEW systems 
and weapon systems, as a matter of 
policy. In other words, MOFE are nec- 
essary. But MOFE are inherently less 
discriminating than MOEs or MOPs. 


e Exising C? and force-level models 
should be used to avoid the necessity of 
building new, unique data bases. (This 
point was contested later by the sponsors 
of the meeting who assured participants 
that resources would be made available 
for the necessary new models and data.) 


3.4 Methodologies for Deriving Mea- 
sures 
The most familiar methodology in the 
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MORS community is the MCES. It is based 
on the scientific method and the cybernetic 
loop model of C?. Although implementation 
is not complete, it is difficult to attack as 
unsound. 


A more mathematically rigorous method- 
ology, an offshoot of the MCES, has been 
developed by Paul Girard, one of the MCES 
originators. The latest version of this ap- 
proach introduces an expected utility frame- 
work to deal with decision-making. It has 
not yet been applied to a real problem be- 
cause of implementation difficulties. 


Another offshoot of MCES has been 
proposed by several TRAC analysts. A read- 
ahead paper by Mr Mike Ingram and Cap- 
tain Robert Short proposed a functional 
breakout based on the Army's doctrine as 
described in FM 100-5. Their methodology, 
called Structured Resolution Approach, 
describes three levels for analysis of com- 
mand and control systems. A mission level 
examines measures that determine if a force 
was able to accomplish its mission. This 
level generally focuses on command and 
control measures of effectiveness. An 
airland battle tenet level examines the ability 
of a C? system to support the four tenets 
(agility, depth, initiative, synchronization). 
This level focuses on command and control 
measures of effectiveness. Finally an imper- 
ative level focuses on the ability of a C* 
system to contribute to implementing the 
imperatives of AirLand Battle doctrine 
(unity of effort, anticipation, concentration, 
main effort, press the fight, move fast, use 
terrain, conserve forces, joint and combined 
arms, morale). This level relates to the 
MOE/MOP for C? systems. The Structured 
Resolution Approach shows promise for 
Army C’IEW analysis, but may not apply to 
other services. 





This approach has recently been modi- 
fied to examine C*IEW systems in terms of 
their contributions to the battle. Under this 
modification, C’IEW systems are examined 
to determine their ability to contribute in 
various areas where a C'IEW system is 
expected to enhance the commander's execu- 
tion of the battle. Within each area of con- 
tribution, measures are identified which de- 
scribe the amount of contribution being 
made by the CÍTEW system. Example areas 
of contribution are shown in Table III-2. 


Another methodology briefly discussed 
was the Headquarters Effectiveness Tool 
(HEAT) which is now in use in the Army as 
a part of the Army Command and Control 
Evaluation system (ACCES) for exercise 
evaluation. ACCES has a cybernetic loop 
like the functional process loop in Module 3 
of the MCES. It has an overall MOE of 'life 
of the plan’ and a set of MOE/MOPS at 
each stop in the cycle. They compare ob- 





servable performance with the standards 
derived from standard operating procedures 
(SOPs). HEAT has a proven data collection 
scheme for headquarters processes in field 
exercises or experiments. 


A final methodology is the new frame- 
work for C? within the Combat Science 
definitions devised by a group of prominent 
analysts. The C? definitions have been sug- 
gested by Professor Wayne Hughes in his 
paper C in a Theory of Combat Science 
(NPS Technical Report No. 55-89-05). This 
paper draws heavily on the MCES and 
HEAT methodologies, but is consistent with 
the broader scope of Combat Science. 


Of the methodologies, to date, the re- 
view of literature and analysts’ experience 
suggests that the MCES had the widest sup- 
port as the broadest, most applicable (but 
incomplete) methodology. 


Table 111-2. Areas of Contribution for a C'IEW System 


Develop situation awareness 


Provide a common picture of the battlefield 


Facilitate control of battle tempo 


Reduce shooter to shooter timeline 


Filter information 


Allow anticipatory actions 


Enhance commander's agility 


Support synchronization of the battle 


Support commander's initiative 


Provide near real-time intelligence 








Chapter IV 


Quantifying CIEW Measures 


4.0 Current Capabilities 

There are three current efforts within the 
Army to produce quantification of C’IEW 
measures. The first is the Command and 
Control Functional Area Model (C?FAM) 
effort to support the TRAC command and 
control studies. It incorporates a Model- 
Test-Model approach and relies on exercises 
and subject matter expert (SME) panels for 
C? input data. SME panels use a structured 
Delphi technique to capture data. 


The C?FAM approach uses command 
and control performance models linked to 
force effectiveness models to determine the 
effectiveness of a C? system. The perfor- 
mance models are Petri net representations 
of information flow within the C* structure. 
The goal of this modeling effort is to repre- 
sent all battlefield functional areas (BFA) 
from theater to brigade level. The BFAs 
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include maneuver, fire support, intelligence, 
air defense and combat service support. 
These performance models are then linked 
to an appropriate force effectiveness model 
such as Vector-in-Commander (VIC), Eagle, 
or TACWAR. This linkage provides the key 
to demonstrating the combat effectiveness of 
the C? system. By using the output from the 
performance models as input to the force 
effectiveness models, the C? system's impact 
on battle outcome can be determined. This 
methodology allows the conduct of a perfor- 
mance analysis as well as an effectiveness 
analysis for the C^ system. Figure IV-1 
provides a graphical representation of the 
C?FAM methodology. 


As seen in the figure, the methodology 
incorporates a modeling tool called Modeler 
to develop the C^ performance models. 
Modeler allows the rapid creation of Petri 
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Figure IV-1. C'FAM Methodology 
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Table IV-1. OPVIEW Methodology 


OPVIEW 
(Operational Value of Intelligence and Electronic Warfare) 


e OPVIEW focuses on MOEs for Intelligence (and EW somewhat) as they relate to the command and 


control decision process. 


€ More specifically, the emphasis is on mission accomplishment, support of the plan and monitoring the 


plan through PIRs (prioritized intelligence requirements). 


€ The methodology is based on PIRs. 


e Any PIR can be classified as a combination of eight functions: 


= Detect 

Locate Generally 
Locate Precisely 
Classify 

Identify 

Trade 

Acquire 


= BDA (assess remaining operational capability) 

Intelligence collection assets attempt to meet PIRs. 

Meeting (or failing to meet) PIRs in a timely manner provides a perception ta the command element. 
The perception provides information to the plan status/accomplishment. 


Plan status affects the decision process. 
The decisions affect the forces. 


O00 0 0 


© Orders to the forces affect the battle outcome. 


e EW supports these eight PIR functions, and counters the enemy's ability to support his own eight PIR 


functions. 


e This methodology has been applied to noncombat as well as combat missions. 


e The methodology has been applied in three model applications. 


nets to represent the information processing 
in a command and control structure. Mod- 
eler was developed by Alphatech, Inc. for 
the Air Force, but it is now being used by 
numerous military and civilian organiza- 
tions. 


The second approach is employed at the 
ATCCS Experiment Site (AES) located at 
Fort Lewis, Washington. In this approach 
the activities of the 9th Infantry Division 
(ID) headquarters have been extensively re- 
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corded and analyzed in field exercises. In 
this effort there are a number of headquar- 
ters tents and shelters which are instrument- 
ed with TV-recorders. Additionally, a data 
capturing system taps into all data bases in 
the headquarters. This effort has developed 
a cadre of experienced analysts with an 
understanding of the process flow within a 
command post. Unfortunately, the 9th ID 
has been deactivated, but part of the 7th ID 
will move to Fort Lewis and the AES will 
probably continue. 


A third effort is the RAND Operational 

Value of Intelligence and Electronic Warfare 
(OPVIEW) methodology. This process is 
now in development as three related models 
which are based on the human translation of 
Prioritized Intelligence Requirements (PIRs) 
into intelligence taskings. The results of 
collection activities are displayed in a series 
of simple plots. Table IV-1 describes the 
approach in some detail. It is an aggregate- 
level approach which may not have suffi- 
cient detail to represent C’IEW. 
4.1 Problems with Current Capabilities 
The existing capabilities have several 
major problems. In developing a capability 
to measure C’IEW, it is difficult to identify 
and structure the decisions involved in or 
impacting on force-level campaigns. The 
scope and amount of detail required in the 
various headquarters areas (intelligence, 
operations, logistics, and EW in addition to 
other supporting services) is not clear. There 
is no complete doctrinal set of the decisions 
at the joint level. As LTC Shoffner said in 
his presentation (see page 31), the US ap- 
proach to command is more parallel and 
intuitive than the Soviet cybernetic model. 
Service SOPs focus on procedures for plan- 
ning rather than control decisions. 


Modeling of information flows to sup- 
port decisions is weak. C? is often modeled 
by message traffic or message size (bytes) 
alone. While this type of modeling is good 
for communications capacity analysis, this 
approach cannot measure the content of the 
messages nor directly determine changes on 
the basis of the information. The informa- 
tion requirements of some decisions are not 
easily established, in particular, the more 
creative command decisions. | 


There is a poor match of data required 
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within our models. Values for performance 
characteristics that are required for direct 
measurement or in models to produce effec- 
tiveness measures are very hard to obtain 
under the conditions in which they are 
collected/evaluated. 


There is a limited capability to collect 
and quantify C’IEW measures at the cam- 
paign level. Collection opportunities are 
limited simply because of the number of 
exercises that involve the campaign level. 
When exercises are conducted, there ia an 
inherent difficulty in seeing real C'IEW, 
because the exercises tend to be scripted, 
designed for training etc. 


There are several problems associated 
with the data collection process. One is that 
there is often limited access to tests and 
exercises. Units conduct tests and exercises 
for purposes that are often at odds with the 
data collection process. Then there is a com- 
mon purpose, unit commanders often do not 
want additional ‘evaluators’ in the area for 
fear of having their units judged by another 
set of eyes. Even if data were to be collect- 
ed, it would consist primarily of what was 
done with perhaps some insight as to why. 
It would require much more extensive in- 
quires (and be more intrusive to exercises) 
to determine what potential options were 
considered but rejected. 


Data collection is currently performed on 
an ad hoc basis. In most cases the collection 
process uses subjective methods, nominal 
sources and focuses on tests and exercises 
under some artificial conditions. 


Another problem is the lack of collection 
resources. Other than organizations such as 
AES, which is formally structured to collect 
data, most analytic agencies must depend on 

















internal resources to collect C’IEW data. 
Data reduction would also be very manpow- 
er intensive. 

4.2 Solution to Problems 

There are some solutions to the problems 
mentioned above. First, continued automa- 
tion of training and collection devices should 
improve the data collection capability and 
our understanding of what to collect. Sec- 
ond, there must be more top down incen- 
tives for organizations to cooperate and 
support a data collection effort. Finally, the 
communities and services need to develop 
common data structures for C’IEW that will 
meet the needs of all. 


There is clearly a need for research, 
developer, analysis, training and testing 
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communities to work together to resolve 
some of the issues related to C’IEW mea- 
sures. As funding and resources become 
scarcer, the opportunities to conduct individ- 
ual community data collection efforts will 
disappear. It is imperative that all communi- 
ties are allowed to participate in every data 
collection opportunity that occurs. This 
means that the organizations that control a 
data collection opportunity must have the 
proper incentive to allow outside organiza- 
tions from other communities to participate. 
Training units must have a good reason to 
allow additional observers into their training 
exercises. Interservice data collection and 
evaluation of same will undoubtedly open all 
kinds of sensitivities. 





Chapter V 


Tools and Methodologies for C’IEW Evaluation 


5.0 Deficiencies in Tools and Methodol- 
ogies 

The deficiencies associated with our 
current tools and methodologies are signifi- 
cant. Our current tools are too manpower 
intensive. We must rely on a multitude of 
observer/controllers to perform the data 
collection for CÍTEW evaluation. The nature 
of the command and control process limits 
our ability to use automated data collection 


for every aspect of the process. 


Although we must perform C’IEW force- 
level analyses to show the effect of C’IEW 
systems on the outcome of battle, our force- 
on-force models that are supposed to support 
these analyses are limited in their explicit 
representation of the C^ process. Most of 
these models rely on a man-in-the-loop to 
capture the initiative aspect of the command- 
er and to perform the command and control 
process. Adding the man-in-the-loop to 
simulations essentially nullifies any ability to 
replicate a run and it normally lengthens the 
run time. 


Scenario dependence in these simulations 
is very high. In most cases, the scenarios 
are developed for weapon system evaluation 
rather than C’IEW system evaluation. The 
result is that C’IEW systems do not get 
stressed during the battle and become sec- 
ondary factors in determining the outcome 
of the battle. 


Also, the technology to move from 
system characteristics to force-level results 
is very limited. We have made only limited 
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progress in linking the performance charac- 
teristics of a C’IEW system to battle out- 
come. Capacity bottlenecks are often not 
modeled in higher level models. 


The C? process in campaign level models 
may not be adequate for the entire purposes 
of all communities. There is a need for 
linkage of simplified processes in campaign 
models to more detailed examination in finer 
granularity models. 


In addition, there are deficiencies due to 
a lack of coordination across services and 
agencies as well as resource constraints. If 
we have CIEW interoperability problems in 
the real world, can we really expect our 
models to behave any better? 
5.1 Overcoming Current Deficiencies 
We should perhaps explore new theories 
of combat and the supporting processes, 
i.e., C? in combat science stochastic decision 
models. This includes applying of other 
methodologies, such as chaos dynamics and 
fuzzy logic to the C’IEW problem. 


Several changes are needed to place 
C’IEW systems on a comparable level with 
weapons systems. Scenarios and models 
need to be developed or extended to specifi- 
cally support C'IEW evaluation. Policy 
makers in DoD must re-examine current 
acquisition requirements to determine if 
changes are need to accommodate the differ- 
ent nature of C’IEW systems. There needs 
to be a better link between the Mission 
Needs Statement (MNS), Operational Re- 











quirements Document (ORD), COEA, and 
the Test and Evaluation Master Plan 
(TEMP). 


We should strengthen our joint efforts 
such as the National Training Center, Na- 
tional Test Bed, and Defense Modeling 
Simulation Office (DMSO). We need to be 
able to respond to pressure for standardiza- 
tion, verification, and validation procedures 
from Congress, GAO, and users. We need 
to explore new computer tools such as 
object-oriented models, simulation languages 
with libraries of objects, and menu-driven 
model building similar to the CAD/CAM 
process. Finally, we may need to develop 
new models of C*, perhaps in the form of 
Petri nets, conditional possibility, and vari- 
able resolution modeling and use these as 
providers of data to our more traditional 
attrition process models. 


There are several opportunities that need 
to be explored as sources of data to support 
models. These include: 


1. Leavenworth training - Battle Com- 
mand Training Program (BCTP) and Tacti- 
cal Commander Development Course 
(TCDC), and senior leadership conferences 
with 1-4 star generals. 


2. National Training Center - ARI, 
Monterey has permanent data (video, graph- 
ics, audio, and hard copy). 


3. ATCCS Experimentation Site - AES 
maintains complete video documentation of 
HQ exercises. 


4. Battle Force In-Port Training (BFIT) 
- Battle group level, simulation driven with 
HEAT data collection. 
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5. FLAG Air Force C* exercises with 
some linkage. 


6. Wargaming at service schools. 


7. Future - Embedded training in every 
weapon system now required. This will 
require a new community of officers with 
expertise in simulation. 


52: 
ologies 
The working groups discussed a wide 
range of possibilities, but had limited suc- 
cess with obtaining closure, except concern- 
ing the desirability of the Model-Test-Model 
approach. Five recommendations are: 


Improving Our Tools and Method- 


l. Develop a mechanism for coordina- 
tion among the various communities and 
services. 


2. We need an orchestrated set of en- 
hanced tools to meet the C’IEW evaluation 
needs. This includes: 


e a means to elicit expert opinion; 


e constructive, virtual models and 
simulations that treat C? explicitly; 


€ participation in live exercises, e.g., 
NTC. 


3. Usethe Model-Test-Model approach. 
4. Develop a special proposal for ad- 
dressing a specific system — a joint system 


is preferred. 


5. We need a vehicle to disseminate our 
progress. 





Chapter VI 


Findings and Recommendations 


6.0 Findings 

The workshop produced several major 
findings. Although most of these do not 
address the original questions associated 
with the objectives, the findings do highlight 
the fact that there is still a long way to go to 
integrate the efforts of the various commu- 
nities to allow them to work together to 
achieve mutually supporting objectives. 
These findings also reveal the need to revi- 
talize the work in this field to provide a 
common, acceptable terminology across 
services and communities. The findings can 
be categorized into five areas: processes, 
measures of merit, methodologies, tools and 
community communications. 


Processes 
The modeling efforts need to focus on 
the C?I processes that can adequately be 


represented. 
The C?I process can be decomposed into 
two parts: automatable and 


non-automatable. Automatable processes 
include information processing, situation 
development, control, communications, and 
concept evaluation. Non-automatable pro- 
cesses include concept formulation, building 
and updating perceptual models. We have 
the capability to represent automatable 
processes in our models, but we must cur- 
rently rely on human intervention to repre- 
sent and perform non-automatable processes. 


The impact of EW must be considered 
in analyses and concepts of operations. 

We saw clearly in Desert Storm the 
effect of EW on the success of our opera- 
tion. It was a significant combat multiplier 
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when used properly in the concept of the 
operation. However, for the most part, the 
overall, joint effect of EW on force level 
operations has been ignored in our 
force-on-force models. We need to improve 
our representation of EW to show its impact 
on shortening the planning cycle time and 
adding to the confusion on the battlefield as 
it degrades the opposing commander’s situa- 
tion awareness. 


Measures of Merit 


Note: The term Measures of Merit 
(MoMs) was introduced to subsume all of 
the measures that characterize a C’IEW 
system's performance (i.e., Measures of 
Performance), its functional effectiveness 
(i.e., Measures of Effectiveness), and its 
impact on force effectiveness (i.e., Measures 
of Force Effectiveness). 


Measures of Merit are defined by the 
level of analysis and the context in which 
they are measured. 

The context in which we are measuring 
MoMs affects the way in which we define 
them. Depending on our perspective, a 
measure might be a measure of performance 
or a measure of effectiveness. It depends on 
the question being answered in the analysis. 


MoMs have the attributes of a name, 
category, system reference (boundary), a 
function reference (purpose), units of 
measure, value measured, and threshold 
value (goal). 

Although this might not be an exhaustive 
set of attributes, it is a good start in the 
process of defining and describing MoMs. 








Additionally work is required to refine this 
list further. 


There are some major MOESs for 
C’IEW systems that should be considered 
in evaluating the system. 

Based on our experience with C’IEW 
systems in using them in a variety of situa- 
tions, we can identify some major MOEs 
that should be considered. These include the 
ability to: maintain multiple views of ene- 
my courses of action; formulate and evaluate 
multiple friendly courses of action; config- 
ure and reconfigure rapidly for new situa- 
tions (flexibility); identify and assess 
'eround truth' and infer from it; and be easy 
to train on and use. 


Alternative C’IEW systems manifest 
greater changes in MOPs than Measures 
of Force Effectiveness (MOFE). 

We should expect that C’IEW systems 
have a small effect on the overall force 
effectiveness. That is, it is not reasonable to 
say that a change in one ratio will win the 
war. We need to be careful and clever in the 
way we evaluate the effect of C’IEW sys- 
tems so that we can discriminate between 
the effects of a C'I system and other factors 
to properly reflect the impact of a C'I sys- 
tem on force effectiveness. However, tradi- 
tional MOFEs at the campaign level should 
not be rejected as a subset of those useful in 
C°IEW analyses, especially when comparing 
sensors and shooters. . 


Methodologies 
There exist some promising, but in- 


complete methodologies to define, derive, 
and measure MoMs. 

We care a great deal about measures, but 
we still need an underlying methodology to 
provide a logical structure for how we ap- 
proach this very complex problem to gather 
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these measures. A lot of good work has 
been accomplished in previous efforts on 
MOEs, but there is still no complete or 
comprehensive methodology that exists. 
However, persistence in linking MOPs to 
MOEs to MOFEs will help in this regard. 


Tools 

Existing force-on-force models do not 
contain modules that adequately represent 
C? processes. 

Current force-on-force models have not 
been very useful for analyzing the full spec- 
trum of C’IEW systems, because they do not 
explicitly represent command and control. 
This limits the Model-Test-Model paradigm 
that is needed to integrate modeling into the 
test and evaluation world. Likewise, the 
theory of how to smartly aggregate lower 
level cause to aggregate level effect needs to 
be extended. 


An orchestrated set of enhanced tools 
is required to meet the C’IEW evaluation 
needs. 

There is no one magic tool that meets 
our needs for evaluating C’IEW systems. 
We need a set of tools that will allow us to 
capture every aspect of the C’IEW process 
to properly analyze a system. These tools 
must include the means to elicit expert 
opinions; the means to capture C? activity in 
live exercises such as at the National Train- 
ing Center (NTC); and, constructive and 
virtual modeling and simulation that treat C? 
explicitly. 


Community Communications 


Continuous  interrelationships are 
needed among the key communities (de- 
velopers, testers, trainers, and re- 
searchers). 


We have learned that people in these 
different communities speak different lan- 


guages and it is sometimes difficult to com- 
municate the needs of one community to 
another. We need to work more closely 
together to find the common ground and 
make our efforts mutually supporting. This 


is true not only within a specific service, but 


across services. 


Security classification barriers need to 
be overcome. 

Historically, we have not done well in 
playing Sensitive Compartmented Informa- 
tion (SCI) systems in our models and simu- 
lations. Very few people have access to 
these systems and information about them. 
If we are to properly represent their role in 
the battle, we need to overcome some of the 
barriers that we now face. 


In the context of a training exercise, 
training and system evaluation goals are 
frequently at odds with each other. 

In training exercises, we have very 
attractive activities with large groups of 
people gathered together, and we are unable 
to take full advantage of an excellent data 
collection opportunity. Although system 
evaluators see this opportunity, those being 
trained naturally view the evaluators as 
additional burdens and hindrances to their 
training. It is also difficult to isolate system 
effects from human training effects without 
the benefit of multiple, independent repli- 
cations. 


Recommendations 
Several recommendations emerged from 
the Workshop. 


6.1 


Clarify the different levels of MoMs. 
More work is needed in this area to 
accurately define the levels of measures to 
be used by all communities. One possible 
way of accomplishing this is through case 
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studies of real systems. Particular emphasis 
should be given to a system which has 
potential impacts across all services, e.g., 
JSTARS. 


Assess the feasibility of existing meth- 
odologies and establish a plan to make 
them more credible and easier to apply. 

It is clear that in this area there are no 
cook book solutions. We need to resist the 
urge to be overly mechanistic. The analysis 
of C’IEW systems will require creative and 
innovative thought. Any approach taken 
should be tailored to bridge the disciplinary 
gaps between the operational and the techni- 
cal levels. Single point solutions are inade- 
quate for this type of analysis. Sensitivity 
analyses are mandatory. Documentation of 
these efforts should be made available 
throughout the DoD analysis community. 


Explore the potential value of applying 
functional decomposition techniques to en- 
hance C evaluation. 

One suggestion is to take a dendritic ap- 
proach to the problem. 


Encourage training activities to collab- 
orate early in the planning process with 
T&E to enhance dual use. 


Take early steps to incorporate C? 
evaluation requirements into DIS and 
WARSIM 2000. 

There is an excellent opportunity to 
affect the usefulness of training simulations 
for the testing and analysis communities. We 
must take advantage of this to provide some 
input into the development of these simula- 
tions. 


Continue the trend to explicitly model 
C?I processes in force-on-force models. 
If we are able to do this, it will lay the 




















foundation to do Model-Test-Model work. 
In parallel, develop robust, good enough 
representations for higher level simulations 
that are tuned by more detailed examina- 
tions. 


Develop theories of complex systems 
management and control. 

It is important that we not only develop 
these theories but we must also be able to 
translate them into practical use. Perhaps 
there are lessons to be learned here from the 
civilian communities or international com- 
munities. 


Develop measurable combat require- 
ments, objectives, and tasks to permit the 
tradeoff analyses of C'I and combat sys- 
tems. 


Explore potential role of evolving syn- 
thetic environments on CI analyses. 


Make the workshop results and DIS 
symposium report widely and quickly 
available. 

We need to energize the entire OR com- 
munity to tackle this problem and make 
them aware of the progress that has been 
achieved so far. We need to include a clear 
and concise description of Louisiana Maneu- 
vers and its role in this effort. 


Enhance the exchange of information 
on prospects and work in progress on C? 
evaluation. 
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Focus forthcoming MORS working 
groups on the key issue of representation 
of C? in analysis. 

6.2 Summary 

The challenge in working with C*IEW 
MOES is to overcome not only the curse of 
dimensionality but also the curse of ambigu- 
ity. Care must be taken to interpret analysis 
results to ensure that they are telling us what 
we think they are telling us. 


The workshop renewed an interest in the 
problems and issues related to C?IEW 
MOESs. The challenge for the community is 
to continue where this workshop ended and 
maintain the interest that was demonstrated 
throughout the workshop. We, in separate 
services and separate communities, must 
work closer together to maintain a common 
language, to mutually support one another, 
and to share our knowledge and experience 
to improve our capabilities to support 
C’IEW systems development, testing, and 
training. 


The results of the Workshop has inspired 
at least one participant. Mike Sovereign has 
suggested that NPS host a follow-on work- 
shop to further develop the joint effort 
initiated here. 
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Excerpts from Workshop Dinner Speech 


COL James McDonough 
Director, School of Advanced Military Studies 
US Army Command and General Staff College 


. . . My topic tonight is What is a Cam- 
paign? That is a rather imposing subject. 
On the one hand, it is a formidable chal- 
lenge to speak to all of you ORSA experts 
in what should otherwise be a light and 
easy-going after dinner speech. . . . On the 
other hand, it is intriguing to talk to you 
about my topic, what is a campaign. . . . 
My approach will be to talk about where we 
in the Army are going with our doctrine. 


My credentials for doing this are two- 
fold. Number one is that I am located at 
SAMS (School of Advanced Military Stud- 
ies) and number two, I had some responsi- 
bility in preparing the next version of FM 
100-5, Operations, the Army's keystone 
doctrine. .. . 


.. . I want to relate doctrine to ORSA 
because that is what you do and that is the 
task before us. From whence does our 
doctrine come? That is the beginning of my 
talk tonight. How do you take all that you 
know, all that you have learned in life and 
bring it all together to write a doctrine for 
warfare in the 21st century. What follows, 
then, as a preface to the doctrine is the 
intellectual odyssey that gets you there. 


Let me begin with my own beginnings in 
this process. . . . To me the big event in my 
young life, the recent event, was World War 
II. It was my focus of warriorship as a 
young man. My father was a veteran of 


29 


World War Il. . . . I admired him, wanted 
to be like him. And I watched all the great 
Hollywood war movies — A Walk in the 
Sun, a gripping display of infantry soldiers 
fighting in Italy; Twelve O'clock High, the 
application of airpower to warfare and the 
brave men that fought the Battle of Britain 
and took the campaign to the Germans, Run 
Silent, Run Deep, the thrilling undersea 
battle taking the fight to the Japanese. From 
all those reflections, I drew this idea of 
heroes. We won our war because we had 
heroes. ... 


When I became a little bit more mature, 
I began to understand that leadership had 
some manifestation on how to use those 
heroes. The great names of my era started 
to become known to me: Dwight D. Eisen- 
hower, President of the United States, Su- 
preme Allied Commander, five star gener- 
al; George C. Marshall of the Marshall 
Plan, architect of the war who would later 
become Secretary of State and Secretary of 
Defense. Vinegar Joe Stilwell walking his 
way out of Burma and telling the press we 
took a licking, we have got to figure out 
what went wrong and go back and return the 
favor — and we did. Omar Bradley, the 
soldier's general; Hap Arnold, founder of 
the Air Force; and Bull Halsey, terror of the 
Japanese, Chester W. Nimitz rises to the 
Occasion, wins our great sea battle during 
the dark days of 1942, when our seapower 
seems to have been destroyed; Douglas 





MacArthur, a man of vision and a perennial 
general. The thought entered my mind we 
won because we had the right leaders not 
just heroes and soldiers. 


A short time later, somewhere in there, 
I remember reading a story about a baseball 
team at Fort Benning in the early 1930s. 
The composition of that team was remark- 
able. There was George C. Marshall in the 
stands, Patch in left field, Gerow at short 
stop, Bradley coaching, Devers at first base 
and on and on it went. On that team of nine 
players practically every one of them rose to 
a three or four star general before World 
War II was over. The thought occurred to 
me that it is not just soldiers and leaders, 
but that it was the richness of our nation in 
materiel and human resources that brought 
us victory. ... 


A little later in life, my youth behind 
me, I went to MIT. I began to run into 
some of the defense intellectuals of the day. 
They had a different way of looking at it. 
Some of you can, I am sure, identify with 
them. You are contemporaries and protégés 
of them. George Rathgens who argued 
adamantly that an air defense system, the 
ABM system, would not be at all effective 
in defending the United States against a 
strategic nuclear attack. He was opposed just 
as adamantly and just as convincingly by 
Albert Wohlstetter and the difference be- 
tween them was really an assumption of the 
probability of launch and successful separa- 
tion of the warhead from the booster rocket. 
The difference between their original as- 
sumptions was only 0.2 as I recall it, but it 
made all the difference in the world at the 
far end as to whether or not the US could 
successfully be defended by a missile sys- 
tem. 
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William Kaufmann, one of the original 
whiz kids" one of the first at RAND (and 
RAND is here tonight), taught me the busi- 
ness of quantifying military resources and 
calculating their military effects against 
likely opponents. His studies of conventional 
and nuclear warfare were very quantified, 
very deep, very provocative. These men, 
and others like them, were saying there are 
ways to think about all this, that if you 
study the piece-parts that can be applied, 
they will tell you how to put together the 
best program, the best expenditure package, 
the best technology to ensure victory as was 
planned. The scientists, the nuclear theo- 
rists, the systems analysts were all suggest- 
ing a method of determining How Much Is 
Enough in military preparedness. 


Herman: Kahn, Charlie Hatch, Alain 
Enthoven led me to a study of Robert 
McNamara and his roots in World War II, 
how he applies his Harvard-learned skills of 
systems analysis to the 8th Air Force in 
Europe and later on to the 20th Air Force in 
the Pacific, then on to the Ford Motor Com- 
pany. He rises to be the Secretary of De- 
fense and he institutes the PPBS system by 
which we fight the war in Vietnam and 
structure much of our defense business after 
the war, a system based on the belief that 
you can look with precision and find the 
critical node, analyze it, vary it and measure 
the outcomes. P. M. S. Blackett is his 
intellectual godfather whose classic studies 
of war talks about the relationships between 
tactics and strategy and uses as the basis for 
his ideas the measurement of the effective- 
ness of air.and anti-submarine warfare in 
World War II, and how best to apply the 
resources of war. My store of information 
on warfare was now starting to get quite 
large and unwieldy, indeed. 


Finally, in my thinking 1 show up at 
SAMS, encouraged to do so by the prescient 
Huba Wass de Czege. He, as you know, 
was so instrumental in the rediscovery of 
operational art and the convincing of our the 
Army that we needed to work on that. From 
him and through SAMS, I begin the studies 
of theories of war that run the gamut from 
Clausewitz, Jomini, Sun Tzu, Mahan, 
Douhet, Liddell Hart and Van Creveld, and 
the Russians Svechin and Tukhachevsky. At 
this point, one starts to come to grips with 
the reality that war is a very complex busi- 
ness, that after all the study of war is just a 
jumble, just like war itself; but out of it 
must come order. You have to study the 
parts but integrate it into the whole. 


When we took upon ourselves the task of 
writing the Army's doctrine we understood 
from the get-go that forecasting the way of 
war was the most difficult thing in front of 
us, that whatever doctrine we devised would 
live in a sea of uncertainty — the uncertain- 
ty of whether or not we had it right, and 
that a wrong guess could result in calamity. 


Michael Howard at Yale advises — and 
we took his words to heart — that the best 
you can hope for is not to have it so terribly 
wrong that you lose at the onset of the first 
battle. That's not a very optimistic view 
point. As we began the process of redoing 
our doctrine, we hoped to do better. Wheth- 
er we did or not remains to be seen. 


My message tonight as I take you into 
the business of doctrine is threefold: the 
first thing is to recognize that warfare — 
understanding it, planning it out and fighting 
it victoriously — is both an art and a sci- 
ence; the second thing is that we chose to 
describe how we as an Army thought about 
war, not prescribe how we expected every- 
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one to fight it in detail; the third is that your 
particular skills of operations research and 
systems analysis can help in improving our 
application of the tools of war in specific 
instances, but that even then it remains an 
uncertain business, affected by the so many 
intangibles the Clausewitz defined as fog 
and friction and chance. 


. . . What are the major points of our 
operational doctrine? What follows is just 
really a surface treatment of the major is- 


Sues. 


The first thing is the idea of the com- 
pression of the strategic, operational and 
tactical. Although there are different levels 
of war, the line between them are fine and 
not all that clear. Any commander must 
realize that he transgresses from one to the 
other in short order if the conditions are 
right. This has major ramifications on the 
way we fight. Second, we need to define the 
end state at the beginning — not at the end. 
You need to know where you are going 
before you start. Third, from that, comes 
the understanding that the campaign doesn't 
end with the last shot. You begin a cam- 
paign in order to accomplish strategic ends. 
Often those strategic ends are not really 
anchored until you put into effect operations 
that will continue after the end of war. 
Fourth, you can not assume yourself at the 
point of the struggle — the locale of the 
campaign — at the start. That was a conve- 
nience of our doctrine during the Cold War. 
We assumed ourselves present. We would 
fight the war in Europe and we were there 
in force at the start. Forward defenses was 
our strategy, it was our watch word. We 
had over 300,000 Americans in uniform in 
Europe, even before any fighting began, 
matched by up to 60 days of supply. Force 
projection is the watch word now and with 

















1t a number of difficult tasks. We will have 
to get ourselves to the contested region, 
along with virtually all our supplies. That 
will be a difficult and challenging business. 
I think ORSA could very definitely apply 
itself there. How do you get there, when, in 
what order, and for what effect. 


That leads quite directly to the next 
concern, integration of all types of war into 
a keystone doctrine. Not just a conventional 
war, but all kinds of war: the conventional, 
the unconventional, the nuclear, weapons of 
mass casualties — all of these can coexist 
simultaneously in the same theater of opera- 
tions. One can flow into the other and they 
must all be addressed in a holistic manner. 
That should be done in our keystone doc- 
trine. 


We take into account that not all we do 
is warfighting, that we, the Army, often find 
ourselves in operations other than war. 
These operations are not unimportant. They 
do not degrade our central focus on 
warfighting. In fact, they are related to 
virtually any campaign you can concoct. But 
they are also important in and of them- 
selves. They must be understood and the 
doctrine must be there to help us think about 
It. 


In one of our most central concerns — 
and I think innovative thoughts — for devis- 
ing doctrine, we came to terms with under- 
standing that there is an American way of 
war, one that is uniquely American, and 
that, therefore, both constrains us and brings 
us the opportunity to think about war in 
certain ways. For example, one outcome of 
that is the desire to achieve quick, decisive 
victory with minimal casualties. 


We related this idea of an American way 
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of war to a world of tighter communica- 
tions, a shrinking globe where CNN was 
ever present. We must imbed in the mind of 
the commander that once he initiates his 
operations, the sands of time are running 
through the hourglass; he is being observed, 
in ways that are unknown to him. Whatever 
action he takes will have political, and there- 
fore strategic, consequences in the homes of 
the voting public. Consequently, it will have 
an impact on the conduct of our campaign. 


We entered into at least the first draft 
with the idea of versatility as a tenet, adding 
to the original four of initiative, agility, 
depth, and synchronization. It was not 
enough to be agile, to shift within the the- 
ater of operations from one mode of fighting 
to the other. We also had to be able to shift 
quickly between theaters and between types 
of warfare and from warfare to operations 
other than war. We recognized that we 
needed to do all this all with a shrinking 
base, that it had to be done professionally 
and proficiently, without ever losing stride. 
We also attempted to articulate a concept we 
call battle space. Battle space, yet to be 
finely resolved, deals with both the intellec- 
tual concept and the physical space in which 
the commander fights his fight. We have 
likened it to the ring mastery of the master- 
ful boxer who knows within the confines of 
his ring -- his battle space — he must oper- 
ate by using every available asset. He must 
decide when to strike, when to maneuver, 
when to attack, when to defend, and when 
to put in the finishing blow. 


We also tried to come to grips with the 
multi-dimensionality of war. We no longer 
call it just AirLand Battle; in fact that term 
has faded. We have not walked away from 
airland battle but we recognize warfare as 
something a lot bigger than that. It is 


air-land-sea; it is space; it is special opera- 
tions; it is electromagnetic; it is joint and 
combined; it is multi- dimensional war and it 
must all be brought together into one whole. 


Tempo enters into the con of our doc- 
trine and what we mean by it, which is not 
just speed. There are times when you speed 
up and you quicken the tempo; there are 
times when you slow down and you back off 
the tempo. Certainly it is a measure in the 
effectiveness of war and has central impor- 
tance to successful warfighting. We took an 
old concept — deep, close and rear battle — 
and we really thought about that. We rede- 
fined its purpose. We used to say that deep 
battle was germane only so far as its effec- 
tiveness to close battle. Today we believe 
that is not the sole determiner. Deep battle 
must also be related to its effect on future 
battles. The simultaneity of operations 
throughout the depth of the battle became a 
more powerful concept. It is geared not only 
to close battle but to future events as well 
that determine the outcome of the campaign. 
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Finally, in this short list of major points 
of what we did with our doctrine we tried to 
relate technology to doctrine. Although we 
did not use words as high-falutin as this, the 
idea we applied is that there is a symbiotic 
relationship — one affected the other — and 
they had to come together in a sensible way. 
You just do not reach for whatever technolo- 
gy that happens to be out there and overlay 
that existing doctrine. You just do not read- 
just the doctrine every time a new item 
comes down the pike. Both had to adapt to 
the other and from that integrate the rela- 
tionship between technology and doctrine. 
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Excerpts from Keynote Address 


LTG Wilson A. Shoffner 
Commanding General, Combined Arms Command 
US Army Training and Doctrine Command 


Our Army faces many challenges as we 
respond to a rapidly changing world. Per- 
haps our greatest challenge is command and 
control and tactical decision making. My 
purpose today 1s to address these two very 
important topics so you will understand 
where our Army is going, with respect to 
doctrine, and what you can do to better 
prepare our trained and ready Army. 


As we work together to meet these chal- 
lenges, it is helpful to understand how future 
battle dynamics — how soldiers fight on the 
battlefield — reveal our requirements for 
modernizing the force. The battlefield has 
always been complicated. The future battle- 
field will continue to be complex. 


Battlefield tempo is accelerating. Sys- 
tems are more lethal, tactical mobility is 
increasing, range of fires is increasing and 
more information and intelligence systems 
are available. Together, this rapid pace will 
make battles short and violent. Increased 
tempo on a complex and lethal battlefield 
has several implications for decision makers. 
Information may be ambiguous. For sure, 
decisionmaking will have less time for deci- 
sions that will have significant consequenc- 
es. This battlefield will require agile leaders. 


The future requires agile leaders capable 
of acting faster than the enemy — mentally 
and physically. Leaders must be capable of 
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rapidly formulating concepts, planning 
operations, making decisions and pressing 
the fight. Agile leaders who quickly and 
accurately apply the conditions of their 
environment will produce decisive victories. 


Command and control procedures must 
support the agile leaders on tomorrow's 
battlefield. We must be capable of providing 
the enemy at least three ways to die simulta- 
neously. That means effectively synchroniz- 
ing our forces and placing decisive combat 
power at the right place and time. This 
requires simple procedures, unified con- 
cepts, nested intents, commanders who can 
see and understand the battlefield, and com- 
manders capable of concurrently considering 
current and future operations. Calm, compe- 
tent, confident commanders capable of 
bringing order to chaos must understand 
command and control. 


The acronym "C?" seems to suggest that 
command and control is one word. Com- 
mand and control is not one word. Each is 
different and carries with it significantly 
different responsibilities. 


Command is an art. Commanders formu- 
late concepts, visualize a future state, assign 
missions, allocate resources for those mis- 
sions, assess risk and make decisions. Dur- 
ing the fight, commanders see and under- 
stand the battlefield, go to the right place at 











the right time, assess the outcome, and 
anticipate change. Commanders lead, guide 
and motivate. Command is commander's 
business. 


Control on the other hand, regulates the 
functions of the organization and is a sci- 
ence. Control is a more precise means 
through which staffs support their 
commander’s intent and work with other 
staffs. Control defines limits, computes, 
monitors, measures, reports, identifies vari- 
ance and analyzes. This is staff's business. 
Commanders anticipate change, and staffs 
project change. 


While command focuses the organiza- 
tion, control regulates the functions of the 
organization. Command cuts across the 
functional areas at any given echelon. This 
enables the commander to use his intent and 
concept of operations to focus the unit and 
synchronize the battlefield operating sys- 
tems. Control, on the other hand, operates 
within a functional area and operates similar 
to a guild. Control depends upon data and 
information flow up and down the functional 
area in a manner similar to a stovepipe. This 
information flow is necessary, but remember 
the more control the less command. Our 
command and control process requires com- 
manders to balance command and control 
and execute mission type orders. 


This process is significantly different 
from the Soviet troop control model. This 
critical path methodology is normative, 
predictive, rational, and easy to automate. 
However, can we model an intuitive, cre- 
ative, parallel and sequential process where 
leaders are able to act in the absence of 
orders? 


Our decisionmaking process is a continu- 
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ous, intuitive, creative, parallel and sequen- 
tial process — not an event. All military 
operations can be divided into three phases. 
concept formulation, planning, and execu- 
tion — CPE. Planning and decisionmaking 
procedures during a campaign differ from 
those procedures used before a campaign. 
During battle, leaders need a streamlined 
process to effectively make decisions in a 
time constrained environment. CPE 1s the 
three phased continuous process that helps 
commanders during battle. 


Embedded in the CPE process is the 
leader's requirement to continuously build, 
adjust and execute plans. While building 
plans leaders need information that supports 
METT-T [Mission, Enemy, Terrain, Troops 
available-Time] analysis and concept formu- 
lation. Similarly, during plan adjustment and 
execution, information to leaders must 
adequately represent the situation — reality. 
Additionally, leader's reconnaissance must 
provide information that helps leaders make 
adjustments prior to and during execution. 
During each of these phases, leaders need 
about right information in certain areas to 
assist with timely, right decisions. 


Commanders make different types of 
decisions. Commanders determine what to 
allocate, when and where to commit, and 
what, when, and where to engage forces. 
These decisions also differ based on the 
level of war. Tactical decisions are mostly 
decisions to commit, execute and engage. 
These are means, space, and time decisions. 


Commanders have many challenges with 
respect to ACE. In this example, the com- 
mander strives to minimize the delta be- 
tween committed and engaged forces. The 
challenge is to know when to maximize the 
use of force to get overwhelming combat 


power into the fight and change the slope of 
the line. The commander needs the right 
information to help make these decisions. 


Decisionmaking requires relevant, time- 
ly, and accurate information. Not only do 
we need to know what information is re- 
quired, but we must also know where that 
required information will come from. That's 
only one piece of the equation. We then 
need to know how to translate this informa- 
tion into decisions, orders and actions. Com- 
manders can't wait for perfect information 
before making a decision. Even with timely, 
relevant, and accurate information, the prob- 
ability of making the right decision is still 
not guaranteed. Decisions must be decisive 
and based on about right information. Also, 
watch out for the somatotropin impact. You 
know, that hormone which affects your short 
term memory. 


Sleep impacts on decisionmaking. Stud- 
ies have shown that without sleep, task 
performance degrades over time. This is not 
a surprise, but organizing leaders and staff 
to maximize task performance and instituting 
sleep plans is key. The conceptual skills 
required by commanders and like thinking 
battle staffs to generate concepts are the first 
skills to degrade. Leaders need rest too. We 
also must ensure that the right people, type 
and quantity, are in the right jobs. 


The commander has many thoughts but 
the bottom line for success depends on his 
ability to make vital decisions. These deci- 
sions must not only be based on about right 
information, but must clearly define what 
information the commander needs, the 
availability of good information, and a 
means of providing the necessary informa- 
tion. Increase the probability of success by 
using redundant, independent, multiple 





33 





phenomenology systems that are controlled 
and available. 


Commanders require information to 
support decisionmaking during tactical 
operations. Forces accomplish tactical opera- 
tions through moving, setting, striking, and 
recocking. Depressing the probability of 
failure during each of these phases, increas- 
es the probability of success. By identifying 
those events which will cause failure, and 
identifying the information required to 
reduce the risk of failure, we focus on 
what's required to minimize the chance of 
failure. 


While trying to make timely, accurate 
decisions on the current tactical operation, 
the commander is also faced with the re- 
sponsibility of concurrently considering 
possible future operations. His view is 
influenced by his assessment of the probable 
outcome of the current operation. Addition- 
ally, the commander must apply information 
to his estimate of how to best accomplish 
future operations. Information must support 
this requirement to continuously consider 
current and future operations. 


Information collection is a complex 
process. First, consider all the assets avail- 
able to collect. Then look at all the nodes 
the information must successfully negotiate. 
Additionally, the information must get to the 
right person. Given all this occurs, the 
information must still be timely, relevant 
and accurate. 


Let's look at collection requirements and 
consider all the conditional probabilities. All 
of the events listed by the sensor and the 
observable event must occur in order to 
simply say that the sensor is capable of 
detecting salient targets. That's a lot of if, 


and if statements and as you know the prob- 
ability goes down as we work through an 
event. There will be holes in the coverage. 
Holes caused by the sensor and holes creat- 
ed by the event — holes due to masking, 
field of view, resolution or swath coverage. 
HUMINT, in may cases, is the only way to 
fill these Swiss cheese holes. 


Assuming collection is possible, this is 
only one small piece of the kill equation. 
Again, decisions are required based on the 
information provided. Information that sup- 
ports the decide, detect, and deliver method- 
ology will improve decisionmaking. 


What was once a luxury if you had the 
information is now a necessity to successful 
combat operations. Commanders must know 
what occurred over the next hill — they 
must know the outcome of a particular at- 
tack. BDA helps commanders allocate 
means, commit collection assets and execute 
collection plans. These decisions require 
timely, accurate, and continuous BDA. 
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Tempo, complexity, and lethality will all 
increase on the future battlefield. We have 
less time to make decisions, and because the 
battlefield is more lethal, our decisions have 
a greater consequence. The need for infor- 
mation to support timely and accurate deci- 
sions will continue. This is best supported 
by not continuing to try to model our intu- 
itive decisionmaking process, but by using 
expert systems with human interface. Work 
the total system, be ready to fight blind, 
focus on timely, accurate, relevant informa- 
tion, and increase the chance of making 
right decisions. 
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Excerpts from Objective 2 Plenary Address 


Dr. Michael G. Sovereign 
Naval Postgraduate School 


The purpose of this presentation is to 
provide a quick review for you of the at- 
tempts to deal with the problem of identify- 
ing C3I measures of effectiveness. The 
methodology that was developed is called 
the Modular Command and Control Evalua- 
tion Structure (MCES). These efforts are 
certainly not the only efforts that have been 
made in the past, but they do have a MORS 
background and will hopefully get you 
started thinking along this line. . 


I am also going to illustrate the proce- 
dure through an example of a system which 
I happen to be working on right now. It is 
the Vertical Take-off and Landing Un- 
manned Aerial Vehicle (VTOL-UAV). You 
will see right away that it is not at the 
force-level, but rather at the system-level, 
which is more in line with our effort from 
the previous MORS workshop. From this 
work I will try to suggest lessons learned for 
further methodological exploration. . . 


MCES is a seven module process. The 
first module is Problem Formulation. In this 
module the analyst characterizes the decision 
need, what decisions are being supported, 
what analysis is required, what assumptions 
about the problem are required, and what is 
the mission of the system being investigated. 
The scenario in which the analysis will be 
conducted must also be identified here. . . . 
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The second module is the C* system bound- 
ing. In this module, we identify the relevant 
entities, both physical and structural, and the 
boundaries associated with the system de- 
scription. These boundaries include subsys- 
tem descriptions, the C? system, our own 
forces as well as enemy forces that might 
interact with the system. . .. 


The third module is the C? system defini- 
tion. This will include defining the processes 
that fulfill the mission. In this module we 
will also map processes to the system con- 
figuration. For the VTOL-UAV, the pro- 
cesses include sensing, assessing, generating 
the approaches to identify, evade, or trail an 
enemy, selecting mission priorities, plan- 
ning; and directing. . . . 


The fourth module integrates the statics 
and dynamics of the system elements with 
the functions. In this module, we also devel- 
op the relationships between physical enti- 
ties, structures, and processes. This will 
provide a detailed modeling of the in- 
put/output coupling between functions. The 
results of this effort will be an architecture 
in which the system functions. This integra- 
tion of statics and dynamics can be de- 
scribed in terms of entities, structure, and 
dynamics at various levels. For the UAV 
example, it would like look this: 


Level 
Element 


Subsystem 


System 


Architecture 


Entities 
Sensor 


Payload 


UAV & 
Mission 
Plan Center 


Task Force 


The fifth module specifies the measures. 
We will try to describe the measures at each 
of the levels of the onion, from the dimen- 
sional parameters, through measures of 
performance and measures of effectiveness, 
to measures of force effectiveness. This uses 
the definitions from the '85-'86 MORS 
Workshop. The measures that we developed 
for the VTOL-UAV are: 


1. Dimensional Parameters (DP): 


d. 


b. 


Weight, size, power, number of 
channels 

Number of parts, complexity, 
operational modes 


2. Measures of performance (MOP) 


d. 


b. 


Payload, range, endurance, 
speed, altitude 
Probability of detection, sweep 


width 


. Pointing accuracy, navigational 


accuracy 


. Attrition rate, mission success 


rate (no failures) 
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Integration of Statics and Dynamics 


Structure Dynamics 
Search Pattern Sense 
Surveillance & Assess 
Mission Plans Plan 
CIC & NTDs Assess 
OPGENS Generate 
Select, Plan 
Direct 
ASUW Command Assess 
Doctrine & Generate 
OP Order Select, Plan 
Direct 


c. 


Mission planning time 


3. Measures of Effectiveness (MOE) 
a. Sweep rate, detection rate, per- 


cent of targets identified 


b. Improvement in 
awareness 
Efficiency of targeting 


a p 


situational 


Timeliness of tasking of weapons 
Time before enemy detects task 
force 


4. Measures of Force Effectiveness 


(MOFE) 


a. Task force mission success 


b. Mission completion time 


c. Ordnance expended per target 
killed, force exchange rate 


We can look at these measures in terms 
of the structure of statics and dynamics. 











Specification of Measures 


Level 





Entities 
Element (DP) Recognitio 
Differential 
Subsystem (MOP) Pointing 
Accuracy 
System (MOE) Detection 
Probability 
Architecture Force 
(MOFE) Exchange 
Rate 


The sixth module is data generation. 
This module identifies the timelines for data 
generation and how the data will be generat- 
ed — through an exercise, simulation, ex- 
periment, or from subject matter experts. In 
this part of the analysis process, other fac- 
tors related to data generation are discussed, 
such as reproducibility, precision and accu- 
racy, collection process and environmental 
controls. The problem is creating an experi- 
mental design that can deal with scenario 
dependencies and having a man in the loop 
that is not unlike his real command situa- 
tion. ... 


The final module is the aggregation of 
measures and interpretation. This part of the 
process is intended to assemble the results 
and draw conclusions about the data. This 
should answer the basic question — did the 
system provide a positive contribution. 
Some issues that need to be addressed here 
are the scenario dependence of the results, 
causality of measures — invalidated chain of 
events, sufficiency of measures, and inde- 
pendence of trials. 


Despite community acceptance of this 
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Dynamics 
False Alarm Rate 


Structure 
Field of View 


Sweep Width Surveillance 
Efficiency 

Situational Targeting 

Awareness Timeliness 

Task Force Mission Completion 


Mission Success Time 


methodology, MCES has not solved the 
problem of credible C’I evaluations. I've 
come up with several reasons for this, and 
you might be able to add to the list. First, 
the MOP, MOE, MOFE structure is hard to 
explain, even to us analysts, let alone the 
lay person. Second, our ability to model 
information flow for decisions is crude and 
lengthy. We can take an enormous amount 
of time just deciding what is all the informa- 
tion that is needed to make a credible deci- 
sion. Third, we have very few credible 
tools and models for measuring combat 
MOFEs. Fourth, in general, if C^ MOPs are 
doubled in value, the MOEs increase about 
50 percent, and MOFEs increase only 1 to 
10 percent. That's just the nature of the uni- 
verse, in my opinion. Military command and 
control systems are very robust already. 
Adding a little bit more to them doesn't 
really win you wars. This kind of result is 
not looked upon with favor by those selling 
C 


Then how come we did so well with 
Desert Storm? My fifth point is that trained 
people under motivating conditions can do 
wonders even with weak systems. Since we 


don't model on-the-job training, we didn't 
predict this and have a hard time dealing 
with it. Sixth, command and control will 
always exist. The question is how much C^I 
do we add before it begins to get in the 
way. Finally, I believe that training and 
doctrine are the most important parts of C?. 
The rate of introduction of C*I may exceed 
the adaptability of these two. We need to 
keep it simple. 


If that didn't work after six years, where 
do we go from here? We should perhaps 
explore new theories of combat and the sup- 
porting processes, i.e., C? in combat sci- 
ence, stochastic decision models. We should 
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strengthen our joint efforts such as the Na- 
tional Training Center, National Test Bed, 
and Defense Modeling and Simulation Of- 
fice (DMSO). We need to be able to re- 
spond to pressure for standardization, verifi- 
cation, and validation procedures from 
Congress, GAO, and users. We need to 
explore new computer tools such as 
object-oriented models, simulation languages 
with libraries of objects, and menu-driven 
model building similar to the CAD/CAM 
process. Finally, we need to develop new 
models of C?, perhaps in the form of Petri 
Nets, conditional possibility, and variable 
resolution modeling. 








Appendix D . 


Excerpts from Objective 3 Plenary Address 


Dr. Hank Dubin 
USA Operational Test and Evaluation Command 


Looking at C? measures of effectiveness 
from an operational test perspective I will 
begin by identifying some key words within 
the context of basic Army definitions. I will 
follow with some typical measures for com- 
mand and control and discuss how to teach 
C? which might be a way to get a hand on 
some measures and close with presenting 
some challenges and potential problems. 


There are three classes of traditional C* 
measures that we are looking at to test the 
Army Tactical Command and Control Sys- 
tem (ATCCS): 1) tracking information or 
messages that goes around the system 
which is communications, 2) following the 
activity that occurs within the command post 
through the Army Command and Control 
Evaluation System (ACCES), and 3) observ- 
ing the outcomes of exercises such as train- 
ing exercises. . . 


I will now show excerpts of the formal, 
FM 101-5 definitions with emphasis on the 
italicized words. Command and Control is 
a process where forces are directed, coor- 
dinated, and controlled to accomplish the 
mission. In Command there is authority 
vested in the commander to do planning and 
directing of resource. C^ Challenges pro- 
vides for a complex situation which requires 
a decisive edge over an opponent which 
must be achieved and maintained. The C^ 
system must allow for a great variety of 
tasks and missions that the commander must 
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perform that require integration of many 
complex systems. 


The tasks that are to be accomplished 
through the system consist of three compo- 
nents: organization, facilities, and processes. 
C? Organizations include the relationships, 
authority and responsibility of the staff and 
the functional grouping of the staff sections. 
C? Facilities require supporting automation 
and communications systems that are neces- 
sary for effective command and control. C* 
Processes identify the procedures and tech- 
niques used to find our what is going on, to 
decide what action to take, to issue instruc- 
tions, and to supervise execution. Another 
key item is to Get inside the decision cycle 
of the opponent so that faster and better 
than that of the enemy is good. The basis for 
the ACCES methodology is that of consider- 
ing the headquarters as an adaptive control 
system. This provides for directing and 
monitoring the situation and allows the com- 
mander develop alternative or new plans 
based upon how adaptable the plan is when 
faced with a new assessment of the situa- 
tion. Measures of C^ are measured by the 
extent to which the commander's intentions 
are carried out and the ability to cope quick- 
ly and effectively with changes in the situa- 
tion. 


There are also measures of outcomes 
that identify successful accomplishment of a 
specific mission. They indicate observable 





conditions that can be compared with and 
without new capabilities that support a pro- 
cess. The process itself in not needed to be 
known, only the outcome (i.e.. the number 
of fire missions accomplished or number of 
aircraft successfully engaged). Correlation 
of forces and means which is an old Soviet 
concept is the "school house" solution. In an 
exercise situation to identify what is needed 
to get the resources and capabilities in a 
certain position and look at your ability to 
meet the threat... . 


Then there are some intermediate types 
of measures that compare to ground truth. 
It's an assessment to see how well we un- 
derstand various factors about the friendly 
forces, the enemy forces, the physical envi- 
ronment, and capabilities. Measures depend 
on context, most are measures of effective- 
ness and are scenario driven. Measures of 
task accomplishment must measure the 
ability to accomplish the mission. It is one 
of perception, if the commander thinks its 
better then they will perform better and it 
does not matter what the "scientists" say. . 


The Army Command and Control Evalu- 
ation System (ACCES) focused on the per- 
formance that can be observed in the com- 
mand post. It assumed that better plans can 
be executed with less need for modification 
and therefore the plans remain in effect 
throughout their intended life. ACCES in- 
cludes effectiveness measures as well as 
measures for accuracy of the estimate of the 
situation. Some of the measures are: 


l. Effectiveness: 

a. Percent of plans remain in force 
unchanged for the intended peri- 
od of time 

b. Median time used to complete a 
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control cycle 
c. Percent of control cycles due to 
major incongruities 


2. Monitoring: 
a. Percent of information require- 
ments for which there are data 
b. Percent of units for which the 
most recent data are within the 
desired time window 


3. Understanding: Percent of briefings 
understood . 


4. Estimating and Planning: 
a. Percent of estimates developed 
by two or more participants 
b. Median time to issue a directive 
c. Percent of correct predictions 


5. Reporting: Percent of information 
requirements reported 


6. Coordinating: Percent of cells in a 
node holding information aboutcide about 
specific units 


7. Network: Percent of nodes holding 
information that coincides about specific 
units 


Measures of communications would be 
tracking messages especially within battle- 
field functional areas to see if the messages 
get to the right place at the right time and 
see if the system has the time... . 


We teach command and control in a 
variety of ways. Through institutionalized 
training at our service schools we use four 
methods that include texts, lectures, exercis- 
es, and tests. With field exercises such as a 
command post exercise (CPX) we teach how 
to do things better through practical applica- 





tion within a simulated combat situation, 
then balance it through professional readings 
and attending professional symposia. We 
give tests to measure what is taught. What 
we measure with these tests is the ability to 
reason, recall facts, understand principles, 
recognize relevant factors. The assessment 
of these tests comes from outcomes that are 
judged with respect to the school house 
solution. . .. 


There are practical exercises designed to 
test the four phases of the decision making 
process. The first phase is to analyze the 
mission. Exercises to identify the purpose of 
the mission, the essential and implied tasks, 
and the intent of superiors would be per- 
formed. ... 


The second phase is to develop facts and 
assumptions. Here we must develop terrain 
and obstacle overlays, identify avenues of 
approach, and consider weather analysis. 
We must also develop doctrinal, situational, 
and decision support templates as well as 
evaluate the logistic estimate. . . . 


The third phase is to develop courses of 
action. Exercises to establish relative combat 
power, array forces, and develop a scheme 
of maneuver will be performed. The courses 
of action will also identify the C^ means and 
maneuver control measures and include an 
operation overlay with a course of action 
statement. ... 


The final phase is to decide on a course 
of action. The exercises for this phase is to 
war game each alternative course of action. 
This would include preparing an appropriate 
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plan of action needed to counter any course 
of action by the enemy. . .. 


During the evaluation of these practical 
exercises we must determine what will be 
measured. It seems reasonable to measure 
the plans and orders with respect to con- 
forming to Army standards and quality 
(consistency, completeness, clarity, and 
executability) as well as project the battle 
outcomes. . .. 


There are three basic C^ assumptions: 
More information is better, Faster informa- 
tion is better, and Greater accuracy of infor- 
mation is better. These assumptions place an 
enormous burden to identify the resources 
that are required to accomplish a given level 
of effectiveness. . .. 


In conclusion, I must present some 
major potential problems that must be faced. 
What do you do to measure intent if the 
intent changes? At a high level the intent is 
more likely to remain constant while at a 
low level intent changes due to external 
conditions. Standard operating procedures 
(SOPs) are personality driven. We must be 
able to evaluate the techniques, procedures, 
and equipment that supports the decision 
making process without evaluating the 
commander's judgment. Variability due to 
the human element is great. Observations 
are made on the smallest organizational unit 
for which the system that is undergoing test 
is organic. Testimonials may be biased by 
experience and background. . .. 
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Appendix E - 


Excerpts from Objective 4 Plenary Address 


Mr. Edward C. Brady 
Strategic Perspectives, Inc. 


. . - I would like to offer a few obser- 
vations about some of the issues you have 
already discussed. 


First, 1 would like to give a different 
viewpoint about the statements that we better 
define requirements, bring users and devel- 
opers together up front earlier in systems 
development testing, and problems with the 
pace of technology, etc. 1 offer you two 
observations: one is that the Japanese con- 
ceived the idea of the Lexus; designed the 
car; put together a manufacturing capability; 
and produced it, tested it, and sold it in four 
years. Most people think it is a pretty damn 
good car. There is something wrong with 
our process when many systems take fifteen 
years in development. While this is well 
known, 1 would argue that not only is con- 
cept development as difficult or more diffi- 
cult, but also it is as lengthy as acquisition 
development. Thus, our situation is much 
worse than much debate would make it 
seem. We spend fifteen years in concept 
development and we spend fifteen years in 
acquisition and testing, so it takes us thirty 
years from when we think we need some- 
thing to when we begin to field the initial 
increments of new capabilities. In addition, 
at least in the C'I area, we almost never 
succeed at fielding a whole system other 
than a simple thing like a message handling 
switch at once. It is important to recognize 
that, whether we like it, we are locked into 
some form of an incremental approach not 
only because of the way in which the tech- 
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nology develops, but also because it is 
inherent in the basic idea of command and 
control and intelligence that we will con- 
stantly revise what we want. We forget our 
own history, I think, and that is why we get 
locked into some of these problems. 


For example, I spent a lot of time four 
years ago traveling around the world, talk- 
ing to many CINC staffs, talking to the 
chief communications people in different 
services, and the J6/JCS. Everyone of them 
assured me there was absolutely no tactical 
military requirement for imagery, NONE — 
based on military objectives, military strate- 
gy, and military desire. Today, the most 
pressing need in many commander's minds 
is imagery. They not only want it, they want 
it real time at tactical maneuver level. . . 
This type of sudden change in requirements 
is the norm. Thus, I don't see how there is 
any hope at all in the kind of arena we work 
in of having any rigid, long-term standing 
requirement for a system — even if we 
understood militarily what commanders 
today think they want to do. That is not to 
say that we shouldn’t put effort into defining 
existing requirements, but rather that we 
also have to find a method that accommo- 
dates fluid requirements, a process that lets 
us implement them in stages, and field 
increments quickly. Any approach that 
doesn’t contribute to such an end will not be 
useful. 


As a second example, when I was at The 





MITRE Corporation, 1 managed through the 
period when we first introduced personal 
workstations. We went through four genera- 
tions of workstations in nine years. . . .We 
started a program that said while we are 
changing what we are buying, we also will 
reprioritize who has what.. . . The Com- 
manders in the field will never be satisfied 
if we don't find a process that works like I 
have described. They will always want 
something different than a few years before, 
they will always say what we are fielding is 
not what they want. They will always say 
we are too slow. They will always have test 
beds off on the side, finding something that 
uses newer technology than the formal 
program, and replacing whatever system it 
Is — MCS, ASAS — whatever system we 
have been developing through a logical 
process. It is just in the nature of it. It is in 
the nature of the underlying technologies. . 
. So, we must find processes which accom- 
modate these inherent dynamics and not sit 
around and wish they were different. 


. . . I would draw your attention to LTG 
Shoffner's constant references to the acceler- 
ation in the pace of battle. What this means 
in terms of C?I systems is constantly revised 
information processing requirements. Every 
time that we can process information faster, 
the commanders will want to change how 
they fight and. ask that we find ways to 
process information even faster. That is the 
nature of our business. . . . AS soon as you 
start to exploit the benefits of automation, 
you change the functionality of the process 
it supports. As soon as you change the func- 
tion, you see new ways to change the re- 
quirement, and you change the way you 
want to fight. . . . As long as these dynam- 
ics are true in command and control, intelli- 
gence, and electronic warfare, we have to 
deal with requirement changes during devel- 
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opment of systems. We have to deal with 
them in parallel with technology changes. 
We have to have a process that supports 
both types of change. 


I would now like to talk a bit about 
MOEs. A lot of you know that I was a par- 
ticipant, along with a lot of people in this 
room, in the work that went on in the early 
'80s which, in my view, largely culminated 
in the MORS workshop report that you were 
given in the read ahead package. . . . This 
effort baselined most of the extant methodol- 
ogy, a lot of models, surveyed what the 
community had and didn’t have. It made 
recommendations of what we should do next 
and began to fund research efforts to push 
us forward. ... 


. . „The most important features of that 
period were that there was a known commu- 
nity of interest, there was an interest and 
desire in funding activities, and some prog- 
ress was being made. You can argue about 
whether it was fast or whether it was slow, 
that is not my point. My point is that those 
involved had an increasing commonalty of 
definitions and an increasing consensus of 
what we had and what we didn't have and 
an increasing sense of where we were going. 


One of the things I read on the plane last 
night coming here was the paper by Ingram 
and Short. While that paper has much to 
commend it, I was struck by what I think 
was the basic factor of that paper. It went 
back to several original works and said we 
needed to take a step forward. The question 
I put to myself was why are we only taking 
a step forward seven years later? What have 
we been doing in between? What has hap- 
pened to the continuity of effort, the willing- 
ness to fund research, and everything else? 
Based on less than comprehensive observa- 


tions, 1 would answer that we have done 
some good things in between, but we have- 
n't done enough. We have lost too much of 
the continuity of effort. This forum is im- 
portant in the sense that it brings people 
back to the community and refocuses us on 
a problem area. 


It is particularly apropos to do this in 
light of the ferment going on amongst the 
military about how do they want to fight, 
what kind of systems do they need, even 
what is their mission. . . . we are seeing a 
resurgence in intellectual thinking about 
combat and our national role in it. The 
pressing need to redefine the role of the 
military in the post cold war era has pro- 
duced a great amount of very challenging, 
exciting, interesting thinking. . . . All this 
provides our analytical community an oppor- 
tunity to plug into and take advantage of this 
ferment, regain our former focus and em- 
phasis on measures of effectiveness; our 
interest in the theory of command and con- 
trol and how we can experiment, measure, 
and modify architectures and systems. . 
This workshop offers us a good opportunity 
to look at these issues. 


.. . Since I recently have been spending 
a lot of time with the Army and we are at 
an Army installation, I will use some Army 
examples to illustrate some thoughts I want 
to share with you. I don't mean to pick on 
the Army in any way... . At Fort Sill, the 
Army Science Board meeting 1 was attend- 
ing had a briefing on a system called 
ATACMS, which is a long-range Army 
missile. It was a very, very nice briefing — 
excellent graphics, easy to understand, lots 
of computer icons, lots of color — all used 
to portray an interesting message. It used all 
the latest tools in multimedia presentations. 
It also had good analytical substance to it. 
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The major thrust was — so what is the 
case for buying ATACMS? Why should we 
buy it? Of course, the answer was already 
known since it is an Army system we long 
have wanted to buy. It was good advocacy 
analysis. In the course of the analysis, they 
did what you all know is the right thing to 
do — they made extensive use of sensor 
target coverage footprints and timeline com- 
parisons, which then required them as they 
looked at system capabilities to look at com- 
mand and control systems. . . . They used 
very good measures of performance, mea- 
sures of effectiveness and, they did good 
quantification of them with very simple 
arithmetic. 


What I would like to convey to you 
about this study, however, is that we did the 
same study fifteen years ago, the exact same 
conceptual system, exact same sensors, and 
surprisingly enough reached the same con- 
clusions using the same measures. The 
people who worked on this study did not 
know this. My suspicion is . . . that they 
had about ninety to one hundred twenty days 
to quickly respond to a higher headquarters’ 
need to do some analysis that shows the 
utility of ATACMS. They are bright, good 
people. They did a lot of good things in the 
study, but they didn't build on anything that 
had already been done, and they didn't 
know they were reconfirming things that 
already had been analyzed... . 


Another thing that struck me about the 
read-ahead paper . . . is the call for stan- 
dardization. I happen to personally be 
schizophrenic about standardization. I want- 
ed to share this bias with you to help you 
understand what I am about to say. Stan- 
dards help us in many, many ways. How- 
ever, they hinder us in two important ways. 
First of all, once we have standardized, if 


we haven't included everything we need to 
include in our changing environment, : it is 
very difficult to reach agreement to change 
the standards — and so the standards be- 
come inhibitors once we agree to them. 
Second, most people who issue a call for 
standardization, and I may have misunder- 
stood the paper in this sense, seem to be 
saying that if we could just comprehensively 
a priori define all the measures we need and 
standardize them we would not have any 
more problems. . . . To think that we can 
do this is a normal, top-down analytical way 
of looking at things. We need to be working 
on attempts to do so, but to hold up prag- 
matic attempts to move forward in a 
bottom-up way while we are waiting for 
such efforts to succeed would be a crucial 
mistake. We need to move in both directions 
in parallel and accept that we will be doing 
so forever. There are a lot of good things 
we can do now with what we do have as 
measures, what we do have as data, and for 
what we do need to do daily to support the 
various decisions for which analysis is used. 
What we usually fail to do is to build these 
partial efforts continuously into a framework 
that lets us get closer to the whole. Instead, 
we do almost everything incrementally. 
And, do it discontinuously. We do it without 
knowledge of what each other is doing. We 
do not keep integrating and building, and 
stepping back from our efforts and taking 
top-down assessments of where we are. 
Where have we already filled in the blanks? 
What blanks still exist? What new ones have 
come along? 


This brings the discussion back to two of 
the themes I wanted to convey to you — the 
lack of a forum, and the lack of a research 
thrust to what we are doing. While thinking 
about this, I was struck by a talk show I saw 
on TV this morning about the presidential 
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debates. . . . The panel said that the prob- 
lem with the debates is that the questions 
weren't draw up really well. They weren't 
focused enough. . . . They gave too much 
leeway to the candidates. . . An individual 
in the audience stood up and said they didn't 
think that was the problem at all. They 
thought that the problem was the answers 
the candidates gave! I think we ought to 
examine ourselves from that point of view. 
Are we really, in whatever increment of 
effort we are doing, defining rigorously 
what measures we want to use and why; and 
are they, in fact, applicable to the analysis 
we are conducting or the test we are doing 
or whatever decision we are supporting? Are 
they judiciously chosen? Do we know how 
to measure them, and if we know how to 
measure them what data do we need to 
collect to do that? And, if we are going to 
collect that kind of data, what instrumenta- 
tion do we need to collect it? I think we 
need to look hard at the kind of answers we 
would be giving to these questions. There is 
a lot of poor quality work done by us. I 
don't know the reason for that, other than 
maybe time and resources, and a failure to 
use earlier work properly. We have excel- 
lent studies and we have very poor studies, 
and they seem to be done side by side. You 
can go around the community at any given 
time and find this. . . . It isn't unique to 
particular organizations or particular efforts. 
I think we are not hard enough on ourselves 
about what are the answers as opposed to 
what we are missing in the form of ques- 
tions. We don't hold ourselves responsible 
enough for making continuous progress in 
defining measures, using them judiciously, 
and using them consistently — even if we 
don't know what they all are, even if we 
don't have standards for all of them, and 
even if we don't have all the data we need. 
Thus, I think, from a ground-up way of 


operating — we could make a lot of prog- 
ress towards what we could define from a 
top-down need. 


I want to talk a little about technologies 
and tools. . . 


I think that the advances we are making 
in instrumentation are quite phenomenal and 
are very exciting. We could soon collect 
rooms and rooms and rooms of data and we 
may do that. What I am worried about is 
whether we are going to do that to no end, 
or whether we know what we are doing. For 
example, the White Sands Missile Range has 
done a great job of improving instrumenta- 
tion over the last five or six years. The Fort 
Huachuca Electronic Proving Ground is now 
proceeding pell-mell to implement extensive 
automated instrumentation for SINCGARS 
and EPLRS testing — because they were not 
allowed to pursue their instrumentation plan 
of ten years ago. So it is often in the case of 
our business. Not because we do not have 
smart people. Not because they do not think 
ahead about what they will need. But, rather 
because of the way that budgets, approval 
processes, and everything else works — you 
sort of get things just in time rather than 
ahead of time. 


When you get instrumentation just in 
time, then you learn how to use it and de- 
fine what you want to do at the last minute 
— thus, we make a lot of mistakes. Again, 
I think that this is inherent in the nature of 
things. . . . But it is important to remember 
that we can also say — How can we build 
on what we did do? How can we learn from 
what we have now? How can we do the next 
test better? How can we take advantage of 
what the Navy already did, or how can we 
take advantage of what the Army already 
did? We can only improve ourselves in this 
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way through some kind of community forum 
that meets periodically and discusses these 
things. 


There will be a lot of testing going on 
over the next five or six years even though 
we are now buying a lot less. Many major 
C?I systems are just entering early testing, 
many will be going through product im- 
provement, many will be candidates for 
significant technology insertion programs. . 
. . How are we going to know that changes 
being proposed for systems will really im- 
prove it overall as a system? How do we 
know, if we don’t set up an analytical con- 
struct, if we change the communications 
links to do direct fire with imagery to 
ATCCMS what the impact is on the rest of 
the system. We have no practical way to get 
it all out in the field at once and test it. 


This brings me to topics like simulation. 
Many of you know that I have spent a lot of 
time over the last four years on interactive 
simulation concepts. I assume most of you 
know about a DARPA program that they are 
trying to get off the ground called Synthetic 
Battlefield Environment on Demand 
(SYNBED). This is an effort to network 
ranges together, generally in the Southwest 
United States — Nellis, parts of White 
Sands, maybe Twenty-Nine Palms, a Navy 
thing yet undefined, and the Army National 
Training Center. Doing this in a way in 
which the network would be compatible with 
the efforts in the Distributed Interactive 
Simulation area, you would allow a ground 
commander to have an actual battalion in 
attack at the NTC and an air commander to 
have actual aircraft coming in and support- 
ing that battalion with close air support out 
of Nellis; but also have an overall air com- 
mander fighting not only the close air sup- 
port battle but also fighting a whole air 





campaign which is filled in with simulation. 
Similarly, a ground commander would be 
not only fighting the battalion in attack, but 
also fighting a simulated whole division. 
Also, the simulation forces could be paced 
by the battalion that is actually on the 
ground and the aircraft supporting it. In 
addition, one could include all the forces on 
the flank, rear areas, logistical supplies, etc. 


I wanted to bring this up because, as- 
suming that we are going to succeed at 
implementing this effort, we will have a 
great environment rich with C’, intelligence, 
and electronic warfare activities and many 
echelons of command — which we do not 
normally have at our disposal. We will have 
an enormous range of opportunity for imple- 
menting instrumentation. We can practically 
put in data collection probes anywhere we 
like. Much of the activity is simulated, yet, 
we also will have a lot of humans in the 
loop. The question to ask ourselves is are 
we ready to do that? What are we going to 
do with all that data? And, how will we 
know what it means? 


. . . I think the analysts need to get a 
conceptual framework for their work in 
place quickly. We are going to be using 
many new technologies for a lot of things. 
We are going to be using these capabilities 
for training, using them for concept devel- 
opment, using them for system definition, 
electronic prototyping, to carry out part of 
the testing. What are we in our analytic 
community going to do about using all this 
as an opportunity to collect data, to measure 
things, to test hypotheses we have about 
what is better than what, about how we 
know when we have improved command 
and control and intelligence? 


We now have on the battlefield an enor- 
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mous number of high speed, high quantity 
collectors of data — sensors. They are 
dumping data all over the place. What we 
are struggling with at this point is to define 
what pieces of data and information are 
relevant to whom, and where do we want 
data dumped when. We have a lot of tech- 
nology moving into the field very quickly 
that gives us great flexibility in doing this. 
We have a lot of technology becoming 
available that the services will presumably 
buy for training purposes and testing purpos- 
es that will provide us with a lot of data. 


We could gain a lot of insight into many 
issues if we had a broad framework of what 
it is we want to know, and what it is we 
want to measure, and what our hypotheses 
are about those things, and how we can go 
about collecting the data to prove or dis- 
prove those hypotheses, and to quantify 
those measures. I understand that you have 
had a briefing on the art of command and 
the science of control. While I agree with 
the general thrust of this, I certainly don't 
conclude that we shouldn't measure the art 
part of command and control. Our measure- 
ments here may be more subjective, they 
may be more difficult; but it certainly isn't 
something we should stop doing! 


. . .I would like to summarize by empha- 
sizing that my own experience in this com- 
munity, as a participant in combat opera- 
tions, as a national level intelligence analyst, 
and an R&D worker in acquisition, all leads 
me to the same conclusion. We have a pro- 
cess of funding and tasking that is inherently 
not desirable from a logical, analytical point 
of view. We get things at the last minute. 
We are tasked to do things quickly. We 
don't get the resources when we ask for 
them, but rather we get them at the last 
minute. We also have an environment of 


warfighting and technology — both of which 
rapidly evolve. Yet I believe that we can 
make lots of progress if we put our minds to 
it. We can have a forum. We do have some 
amount of money going into research ef- 
forts. We can talk to one another and share 
important information. We can build on the 
many things that are happening. By doing 
more of all of this, I believe we can make a 
lot of progress. 





49 





50 








General Chair: 


Host: 


Technical Chair: 


Deputy Chair: 


Members: 


Site Coordinators: 





Appendix F 


Planning and Organizing Committee 


Mr. Vernon M. Bettencourt, Jr. 
The MITRE Corporation 


Dr. Robert LaRocque 
US Army TRADOC Analysis Command 


Mr. Donald W. Kroening 
US Army TRADOC Analysis Command 


LTC Thomas J. Pawlowski III 
US Army TRADOC Analysis Command 


Mr. J. Douglas Sizelove 

Office of Deputy Under Secretary of the Army (OR 

Dr. Patricia Sanders 

Office of the Director of Program Analysis and Evaluation, OSD 
Mr. Theodore T. Bean 

The MITRE Corporation 

Col Bruce L. Smith 

Air Force Studies and Analysis Agency 


Mr. Steven B. Schorr 

US Army TRADOC Analysis Command 
CPT Peter R. Itao 

US Army TRADOC Analysis Command 


Administrative Coordinators: Richard I. Wiles 


Military Operations Research Society 
Natalie S. Addison 
Military Operations Research Society 


51 





52 











Appendix G 


Working Group Chairs, Co-Chairs, and Speakers 


Dinner Speaker: COL James McDonough 
Director of the School of Advanced Military Studies 
US Army Command and General Staff College 


Keynote Speaker: LTG Wilson A. Shoffner 
Commander, US Army Combined Arms Command 


Group A - Operations, Plans, and Training 
Chair: Lt Col Kenneth C. Konwin 


Air Force Studies and Analysis Agency 


Co-Chairs: Mr. Kenneth W. Bernard 
US PIE Combined Arms Command-Training 

Lt Col Jon Peters 

Joint Warfare Center 


Group B - Analysis 
Chair: Dr. Michael G. Sovereign 


Naval Postgraduate School 


Co-Chair: Mr. Royce Reiss 
Air Force Studies and Analysis Agency 


Speaker: Dr. Michael G. Sovereign 
Naval Postgraduate School 


Group C - Test & Evaluation 


Chair: Mr. James B. Duff 
COMOPTEVFOR 
Co-Chairs: Mr. Robert L. Broderson 


Battelle Pacific Northwest Laboratory 
Dr. Wayne R. Knight 
US Army Option Test & Evaluation Command 


Speaker: Dr. Henry C. Dubin 
US Army Operational Test & Evaluation Command 


53 








Group D - Research 
Chair: | Mr. J. Darrell Morgeson 


Los Alamos National Laboratory 


Co-Chairs: LTC James E. Armstrong, Jr. 
US Military Academy ' 

Dr. Robert J. Might 

George Mason University 


Speaker: Mr. Edward C. Brady 
Strategic Perspectives, Inc. 


54 





Tuesday 
1630-1800 


1800 


1830 


Wednesday 


0800 


0815 
0820 


0900 
0915 


Flexible 


1330 
1345 
1400 
1430 
1445 
1800 


Thursday 


0800 
0815 
0830 
0900 
0915 


Flexible 


1300 
1315 
1330 
1400 
1415 
1730 





Appendix H 
Workshop Agenda 


20 October 

Registration - Frontier Crossroads Club 

Call to Order - Mr. Bettencourt 

MORS Welcome - Mr. Wiles 

Host Welcome - Dr. LaRocque 

Dinner - Frontier Crossroads Club 

Guest Speaker: COL McDonough, Director, SAMS 
Topic: What is a Campaign? 


21 October 

Opening Session - CR 6, Bell Hall 

Charge to the Working Groups - Mr. Kroening 
Introduction of keynote speaker - Dr. LaRocque 
Keynote address - LTG Shoffner 

Topic: The role of force-level command and control in a campaign 
Break 

Working Session for Objective 1 

Lunch 

Summary by Chair, Working Group Á 
Comments from the other Chairs 

Address on Objective 2 - Dr. Mike Sovereign 
Break 

Working Session for Objective 2 

End of Day 1 


22 October 

Summary by Chair, Working Group B 
Comments from the other Chairs 
Address on Objective 3 - Dr. Hank Dubin 
Break 

Working Session for Objective 3 

Lunch 

Summary by Chair, Working Group C 
Comments from the other Chairs 
Address on Objective 4 - Mr. Ed Brady 
Break 

Working Session for Objective 4 

End of Day 2 


55 





Friday 23 October 


0800 Summary by Chair, Working Group D 

0815 Comments from the other Chairs 

0830 Summary from Synthesis Group Chair 

0900 Remarks by Technical Chair 

0930 Conclusions 

1000 End of Workshop 

1015 Executive Session with Working Group Chairs 


56 








Addison, Natalie S. 
Allen, Patrick D. 
Anderson, Charles 
Armstrong, James E., Jr. 
Baer, Dennis R. 

Bean, Theodore T. 
Berenson, Paul J. 
Bernard, Kenneth W. 


Bettencourt, Vernon M., Jr. 


Brady, Edward C. 
Broderson, Robert L. 
Carter, Harvey R. 
Chappel, Oscar A. 
Clark, Lloyd N. 
Davis, David. F. 
Dubin, Henry C. 
Duff, James B. 
Girard, Paul E. 
Goldberg, Edwin M. 
Hager, Ralph S. 
Hollis, Walter W. 
Holt, James T. 
Horner, Mary 
Ingram, Michael 
Itao, Peter R. 
Jennings, Joseph F. 
Kelly, Michael 
Knight, Wayne R. 
Konwin, Kenneth C. 
Kroening, Donald W. 
Kromer, William S. 
LaRocque, Robert 
Martinet, Brian 
Mayer, Hugo E. 
McDonough, James 
McGruther, Kenneth 
Metzger, James J. 
Might, Robert J. 
Moore, Jerry L. 


Appendix I - 
Participants 


Military Operations Research Society 
RAND Corporation 

SAIC 

US Military Academy 

OPNAV 815 

The MITRE Corporation 

TRADOC 

Combined Arms Command-Training 

The MITRE Corporation 

Strategic Perspectives, Inc. 

Battelle Pacific Northwest Lab 

OSD 

Potomac Systems Engineering, Inc 
AFSAA/SAS 

George Mason University 
Operational Test & Evaluation Command 
COMOPTEVFOR 

SAIC 

CECOM 

Lawrence Livermore National Lab 
DUSA(OR) 

OASD (PA&E) 

TRADOC Analysis Command 

TRADOC Analysis Command 

TRADOC Analysis Command 

The MITRE Corporation 

USA Concepts Analysis Agency 

USA Operational Test & Evaluation Command 
AFSAA/SAG 

TRADOC Analysis Command 

BDM International 

TRADOC Analysis Command 

US Army Command and General Staff College 
TRADOC Analysis Command 

USA Command and General Staff College 
USSPACECOM/J3 

OASD(PA&E) 

George Mason University 

Combined Arms Command-Combat Developments 


37 











Morgeson, J. Darrell 
.Pawlowski, Thomas J. III 
Peters, Jon 

Pittman, James T. 
Quensel, Susan L. 
Ralston, Mark H. 
Ratzenberger, Annette C. 
Reader, Thomas W. 
Redwinski, Robert J. 
Reiss, Royce H. 
Riddle, Thomas 
Roberts, John 
Robinette, Stephen 
Sandlin, Alan D. 
Schorr, Steven B. 
Shoffner, Wilson A. 
Sizelove, J. Douglas 
Sovereign, Michael G. 
Starr, Stuart H. 

Van Wyk, Carl 

Visco, Eugene P. FS 
Vozzo, Peter M. 


Westmoreland, Herbert O. 


Wetstone, Shawn C. 
Wiles, John A. 
Wiles, Richard I. 
Youngren, Mark A. 





Los Alamos National Lab 
TRADOC Analysis Command 

JWC 

Potomac Systems Engineering, Inc 
TRADOC Analysis Command 
AMSAA 

USA MISMA 

Army Research Lab 

AMSAA 

AFSAA/SAG 

USACGSC 

The MITRE Corporation 

US Army Command and General Staff College 
NCCOSC RDT&E Div 

TRADOC Analysis Command 
Combined Arms Command 
DUSA(OR) 

Naval Postgraduate School 

The MITRE Corporation 

NAWC, Aircraft Division Warminster 
US Army MISMA 

USA Aviation Center 

Combined Arms Command-Training 
Institute for Defense Analysis 
OUSD(A)/T&E 

MORS 

The Joint Staff, J-8 


58 








ACCES 
AES 
ALSP 
ARI 
ATACMS 
ATCCS 
BCTP 





Appendix J 
Acronyms 


Army Command and Control Evaluation System 
ATCCS Experimentation Site 

Aggregate Level Simulation Protocol 

Army Research Institute 

Army Tactical Missile System 

Army Tactical Command and Control System 
Battle Command Training Program 

Battlefield Damage Assessment 

Battle Force In-Port Training 

Command and Control Functional Area Model 
Command, Control, Communications, Intelligence, and Electronic Warfare 
Computer Aided Design. 

Cost and Operational Effectiveness Analysis 
Defense Modeling and Simulation Office 
Department of Defense 

Echelons Above Corps 

Essential Elements of Analysis 

Electronic Warfare 

Headquarters Effectiveness Assessment Tool 
Modular Command and Control Evaluation System 
Mission, Enemy, Terrain, Troops, and Time 
Mission Need Statement 

Measure of Effectiveness 

Measure of Force Effectiveness 

Measure of Merit 

Measure of Performance 

Military Operations Research Society 
Model-Test-Model 

National Training Center 

Operational Value of Intelligence and Electronic Warfare 
Operational Requirements Document 

US Pacific Command 

Priortized Intelligence Requirements 

School of Advanced Military Studies 

Subject Matter Expert 

Standard Operating Procedure 

Synthetic Battlefield Environment on Demand 
Tactical Fire Direction System 











TRADOC 
USACGSC 
UAV 

VIC 
VTOL 





Tactical Commander Development Course 
Test and Evaluation Master Plan 

Tactical Operations System 

TRADOC Analysis Command 

Training and Doctrine Command (US Army) 
US Army Command and General Staff College 
Unmanned Aerial Vehicle 

Vector in Commander 

Vertical Takeoff and Landing 


60 


